Your MCP server is stateful. Your load balancer doesn’t know that.

Your MCP server works on localhost. It works on a single cloud instance. Then you put it behind a load balancer, scale to two replicas, and sessions start failing in ways that look random: requests that reference a session the server claims doesn’t exist, streams that die mid-response with a 502. Nothing in your code changed. What changed is that your infrastructure assumes a stateless service, and an MCP server over streamable HTTP is not one.

A session is an open stream in one process’s memory

Streamable HTTP looks like ordinary request/response from the outside. It isn’t. When a client initializes a session, the server typically opens a server-sent-events stream back to it and keeps a transport object for that session in memory — commonly a map from session ID to transport. Every subsequent request in that session assumes the process that receives it holds that entry. The stream is how the server pushes notifications, progress, and elicitation requests to the client; the map entry is how a follow-up POST finds its way back to the right stream.

That is server-side state, pinned to one process. Your load balancer doesn’t know it exists.

The second replica breaks it

With one replica, every request lands on the process holding the session. With two, a round-robin load balancer sends roughly half of your follow-up requests to a replica that has never heard of the session. The request fails. The client retries, maybe initializes a new session, and your logs fill with session-not-found errors that correlate with nothing.

The sharpest version of this involves elicitation — the MCP mechanism where the server pauses a tool call to ask the human a question. I hit this on an MCP server I built at ZeroClick that provisioned third-party services inside agent sessions: provisioning would pause on a question, the user would answer, and the answer POST had to reach the exact instance holding the open SSE stream for that session. Behind a multi-replica load balancer it often didn’t. The fix was generated-cookie session affinity at the load balancer — the balancer sets a cookie on the first response and routes every request carrying it to the same backend. Set the cookie’s TTL to match your session lifetime; an affinity cookie that outlives the session pins clients to a backend for no reason, and one that expires early recreates the original bug.

The alternative to affinity is externalizing the transport state so any replica can serve any session. That is more work and covered in the decision guide below. But do one or the other before adding the second replica, not after.

Proxies kill healthy streams

An SSE stream that carries a slow tool call can sit quiet for minutes. Managed load balancers ship with backend idle timeouts tuned for request/response traffic — 30 seconds by default on the setup I ran. When the stream stays quiet past that, the balancer cuts it and the client sees a 502 on a connection that was working correctly. The fix is configuration, not code: I raised the backend timeout to 300 seconds, sized to the longest quiet period a tool call could plausibly produce.

While you’re in that layer, check two related things. Disable response buffering anywhere in front of the stream — a buffering proxy holds SSE events until its buffer fills, which defeats the point of streaming. And confirm your framework actually runs the route on a long-lived server runtime. Static optimization and edge runtimes are built for short requests; a route that holds a stream open needs a process that stays up.

Your in-memory session map is a leak

Clients disconnect without saying goodbye. An agent gets killed, a laptop lid closes, a network path dies — and the clean-shutdown handler you wrote never fires. Each of those leaves an entry in the session map and a transport object nobody will ever use again. Under real traffic the map only grows.

The fix I shipped: sweep the map on a TTL — 30 minutes in my case — and track last-activity time on each session, updating it on every request the session receives. Expire sessions whose last activity is older than the TTL. Last-activity matters more than creation time here: a session created two hours ago that handled a request forty seconds ago is alive, and expiring it would cut off a working client mid-conversation. A session created ten minutes ago that has been silent ever since is probably a disconnected client you’ll never hear from again.

Health checks and draining

A process holding fifty open streams can pass a naive health check while being unable to do useful work, and can fail one while doing its job perfectly. Define healthy as “can accept and serve a new session” — a lightweight endpoint that exercises nothing long-lived — and keep the health check off the streaming path entirely.

Deploys are where session affinity collects its price. Every session is pinned to a process, so replacing that process severs its sessions — and if a tool call is paused waiting on a human, the work in flight dies with the stream. Configure your platform to drain on deploy: stop routing new sessions to the old replica, give existing streams a window to finish or expire, then terminate. The right drain window is related to your session TTL; an abrupt kill turns every deploy into a small outage for whoever was mid-session.

A small polling detail worth stealing

Somewhere in a stateful MCP system you will write a status-polling loop — a client waiting for a long-running operation to finish. Put the sleep at the end of the loop body, not the beginning, so the first status check happens immediately. Operations that complete quickly get their result on the first check instead of eating a full polling interval for nothing. It is a one-line difference and it shaves the common case for every caller.

Decision guide

• Accept sticky sessionswhen you run a small, stable replica count and can tolerate losing in-flight sessions on deploys and scale-in. It is a load-balancer setting plus a TTL sweep — the cheapest correct answer, and where I’d start.
• Externalize session statewhen deploys are frequent, replicas autoscale, or dropped sessions are expensive. Transport state moves to a shared store such as Redis so any replica can resume any session. You pay in architecture: the SDK’s in-memory transport model no longer maps cleanly, and you own the serialization and resumption logic.
• Use a job-and-poll designwhen the real problem is long-running work rather than interactivity. Have the tool call return a job ID immediately and let the client poll a status endpoint — or take a callback. No request holds a stream, every request becomes stateless, and the load balancer goes back to being boring. If your tools don’t need mid-call elicitation, this is often the strongest option.

Whichever you choose, choose it deliberately. The failure mode of not choosing is the default one: in-memory state, round-robin routing, and a bug report that says the server works fine on localhost.