The evolution of AI coding agents has quietly ushered in a new bottleneck: the transport layer. It’s a detail that, personally, I think many developers overlook, but it’s becoming a first-order concern as workflows grow more complex. Let me explain why this matters—and why it’s far more interesting than it sounds.
Imagine you’re on a flight, trying to use an AI coding agent like Claude Code to debug a project. The agent needs to read files, propose edits, and run tests—a typical multi-turn workflow. But with each turn, the entire conversation history is resent over HTTP, ballooning the payload. By the third or fourth turn, requests time out due to bandwidth constraints. This isn’t just an edge case; it’s a symptom of a broader architectural mismatch between stateless APIs and stateful agent workflows.
The Hidden Cost of Statelessness
What makes this particularly fascinating is how the problem scales. For single-turn interactions, stateless HTTP works fine. But agentic workflows involve 10, 20, or even 50+ turns, each retransmitting the full context. By turn 9, the payload can grow to 10x its initial size. This isn’t just about latency—it’s about efficiency at scale. If you take a step back and think about it, this linear growth in payload size becomes a critical bottleneck for both clients and servers.
OpenAI’s WebSocket mode, introduced in 2026, addresses this by caching conversation history server-side. Instead of resending the full context, clients send a reference ID and incremental updates. The result? An 80-86% reduction in client-sent data and 15-29% faster execution times. But here’s the kicker: the benefit isn’t about the protocol itself—it’s about avoiding redundant context transmission. Any stateful approach could achieve similar gains.
Why This Matters Beyond Speed
One thing that immediately stands out is the broader architectural implications. Stateful continuation isn’t just a performance hack; it’s a shift in how we design AI workflows. But it comes with trade-offs. Reliability, observability, and portability become harder to manage. For instance, WebSocket connections are ephemeral and non-multiplexed, meaning parallel tasks require multiple connections. This raises a deeper question: are we willing to sacrifice some flexibility for speed?
What many people don’t realize is that this isn’t an OpenAI-specific issue. While WebSocket mode is currently their competitive advantage, the pattern applies to any system with multi-turn context. The real innovation here is server-side state management, not the protocol. Yet, the industry is fragmented—Anthropic, Google, and others haven’t adopted equivalent solutions. This creates a provider lock-in dilemma for developers who want to switch models.
The Bandwidth Math at Scale
A detail that I find especially interesting is the server-side impact. With millions of concurrent coding sessions, the savings compound. Our benchmarks show that WebSocket reduces ingress traffic by 144 GB per 40-second task at scale—a 29 Gbps reduction. This isn’t just about client efficiency; it’s about reducing load on tokenizers and API gateways. What this really suggests is that stateful designs aren’t just a client-side optimization—they’re a server-side necessity for high-volume workflows.
The Statefulness Spectrum
From my perspective, the choice between stateless and stateful designs isn’t binary. It’s a spectrum of trade-offs:
- Stateless HTTP: Simple but inefficient for long workflows.
- WebSocket + in-memory state: Fast but volatile.
- Persisted state: Durable but slower.
The sweet spot for most agents? WebSocket with in-memory caching. It’s fast, compliant with zero-retention policies, and avoids mid-task recovery complexities. But it’s not a one-size-fits-all solution. For multi-provider setups or stateless backends, HTTP remains the pragmatic choice.
Looking Ahead: Will Standards Emerge?
If you ask me, the biggest question here isn’t which protocol to use—it’s whether the industry will converge on a standard for stateful continuation. Right now, WebSocket is OpenAI’s advantage, but developers want flexibility. Will competitors adopt similar mechanisms, or will this remain a differentiator? Personally, I think we’re headed toward a hybrid future where stateful designs become the norm for complex workflows, but HTTP sticks around for simplicity.
What this really suggests is that transport layers are no longer an afterthought. As AI agents mature, these architectural decisions will shape performance, cost, and developer experience. It’s a reminder that even in the age of LLMs, the plumbing still matters—a lot.
