nginx as a WebSocket reverse proxy: the four knobs that actually matter

The first time I dropped location /ws { proxy_pass ... } into an nginx config and hit refresh, I got exactly nothing back. No error in nginx's log. A green 200 in the access log. And a browser console screaming Error: Unexpected server response: 200 at me like I'd forgotten how HTTP works. I hadn't. I'd just walked into the oldest trap nginx sets for anyone shipping a WebSocket: it treats HTTP as its home turf, and WebSocket is close enough to look ordinary but different enough that the defaults are actively hostile.

There's a second version of this trap, and it's meaner. The handshake works. Sockets open, messages flow, everything is green. For exactly sixty seconds. Then every socket dies at the same instant, the client library retries, and a stampede of reconnections hits the upstream. Both traps are documented in a paragraph buried three pages into the official docs. Both are silent from nginx's own error log. And both come down to the same four knobs.

Let me walk through a five-step repo that starts with a naive config and ends with one that survives 200 concurrent sockets held silent for ten minutes. Each step is a real git commit you can git checkout and reproduce on a laptop. The upstream is a small Bun WebSocket echo server, the proxy is nginx 1.25, and the load driver is k6. Nothing esoteric, nothing paid, nothing that won't run on any Linux box you can ssh into. If you're shipping a chat app, a realtime dashboard, an MQTT-over-WebSocket gateway, or a Phoenix LiveView backend behind a shared nginx frontend, the four knobs in the title are yours: proxy_set_header Upgrade, proxy_buffering, proxy_read_timeout, and keepalive_timeout. Everything else in the article is context around when to turn each one and by how much.

Step 1: a WebSocket upstream we can actually break (commit 71f2d1d)

How do you tell whether a broken WebSocket is nginx's fault or the upstream's? Build an upstream you can trust (one with a longer idle timeout than nginx and a plain HTTP endpoint on the same port), and any remaining mystery lives in the proxy. A dozen lines of Bun give us that:

const server = Bun.serve<{ clientId: string }>({
  port: PORT,
  fetch(req, server) {
    const url = new URL(req.url);
    if (url.pathname === "/health") return new Response("ok");
    if (url.pathname === "/ws") {
      const upgraded = server.upgrade(req, {
        data: { clientId: crypto.randomUUID() },
      });
      if (upgraded) return;
      return new Response("upgrade required", { status: 426 });
    }
    return new Response("not found", { status: 404 });
  },
  websocket: {
    idleTimeout: 300,
    open(ws) { ws.send(`welcome ${ws.data.clientId}`); },
    message(ws, msg) { ws.send(`echo: ${msg}`); },
  },
});

Two design choices matter here. First, idleTimeout: 300 means Bun itself will hold the socket open for five minutes with no traffic. That's deliberately much longer than nginx's sixty second default, so that when a socket does die we can be sure the proxy killed it and not the upstream. Second, /health gives us a plain HTTP endpoint on the same port. If curl http://nginx:8080/health returns ok but wscat -c ws://nginx:8080/ws fails, the problem is definitely upgrade related, not a firewall or a routing mistake.

The Bun install one-liner lives at https://bun.sh/install if you want to run the upstream directly on your host machine, though docker compose up --build starts both containers together and is the path the repo assumes. From there, docker compose exec app wget -qO- http://localhost:3000/health should print ok, and wscat -c ws://localhost:3000/ws (talking to Bun directly, bypassing nginx) should print welcome <uuid>. That last check is the baseline: if the upstream works and the proxied endpoint doesn't, the proxy is at fault.

Step 2: the config that almost works (commit adf4d2b)

A friend once shipped this exact naive config to staging, saw green 200s in the access log for an hour, and only figured out the bug when a browser tab in a coworker's demo went silent mid-typing. This is the config every tutorial starts with, and it's wrong in two specific ways:

http {
  upstream ws_upstream { server app:3000; }
  server {
    listen 80;
    location /health { proxy_pass http://ws_upstream; }
    location /ws    { proxy_pass http://ws_upstream; }
  }
}

Try to connect a WebSocket client to ws://localhost:8080/ws and you get an HTTP 200 back. That is the bug. The client sends its handshake with Upgrade: websocket and Connection: Upgrade headers, but by default nginx speaks HTTP/1.0 to the upstream, and HTTP/1.0 has no concept of connection upgrade. nginx therefore quietly drops both headers on the way through. The upstream sees an ordinary GET request, replies with 200 OK and a small body, and the client's WebSocket library rejects the response because it expected a 101 Switching Protocols.

Nothing about this is logged by nginx at error level. The access log records a normal 200. You'll look at the wrong thing for hours if you don't already know the shape of the bug. The fix has three moving parts and they all need to be there together, which is why the next step is worth its own commit rather than a one-line change.

Step 3: pass Upgrade and Connection through, safely (commit 91d9db7)

What does nginx actually send upstream when a browser asks to upgrade a connection? By default, none of the three headers that matter get through. That's why the fix has three moving parts that must all land together.

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

server {
  location /ws {
    proxy_pass http://ws_upstream;
    proxy_http_version 1.1;
    proxy_set_header Host              $host;
    proxy_set_header Upgrade           $http_upgrade;
    proxy_set_header Connection        $connection_upgrade;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
  }
}

proxy_http_version 1.1 upgrades the upstream conversation. proxy_set_header Upgrade $http_upgrade copies the client's Upgrade: websocket request header onto the outbound request, so Bun can see it. And proxy_set_header Connection $connection_upgrade sends Connection: upgrade when the request actually asked for one and Connection: close when it didn't, courtesy of the map block above the server directive.

Why the map? Because the same /ws location can be hit by a plain HTTP client such as a monitoring probe, a warmup request, or a mistaken curl. If you hard-code Connection: upgrade the upstream sees a nonsense request whose upgrade header is empty and whose connection header claims an upgrade, and it either closes the socket or returns 400. If you hard-code Connection: keep-alive you break real upgrades. The map picks the right value per request. It's the single most-copied nginx idiom on Stack Overflow for a reason, and the official upgrade docs at https://nginx.org/en/docs/http/websocket.html use exactly this pattern.

Reconnect with wscat -c ws://localhost:8080/ws after this commit and you get welcome <uuid> back. The handshake works. Send a few messages, they come back as echo: <msg>. Job done, ship it, right? Then someone opens the app and steps away for coffee. Sixty seconds later every socket dies.

Step 4: keep long-lived sockets alive (commit 4eb7053)

Four defaults kill long-lived WebSocket connections in production. Step 4 addresses all four in one pass because leaving any of them at their default value re-breaks the same class of client.

upstream ws_upstream {
  server app:3000;
  keepalive 64;
  keepalive_requests 10000;
  keepalive_timeout 60s;
}

server {
  keepalive_timeout 65s;

  location /ws {
    proxy_pass http://ws_upstream;
    proxy_http_version 1.1;
    proxy_set_header Host              $host;
    proxy_set_header Upgrade           $http_upgrade;
    proxy_set_header Connection        $connection_upgrade;

    proxy_buffering off;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
    proxy_connect_timeout 10s;
  }
}

Knob one, proxy_buffering off. WebSocket frames are a stream, and nginx's default proxy buffer is an 8 KB block that only flushes when full or when the connection closes. In a chat workload where each message is a few hundred bytes, the buffer might sit half-full for minutes, and the recipient sees traffic arrive in synchronised bursts every time some unrelated request finally fills a buffer. Turning buffering off makes nginx forward each frame as it arrives, which is what you want for a realtime protocol.

Knob two, proxy_read_timeout 3600s. This is the ceiling on how long nginx waits for a byte from the upstream before killing the socket. The default is 60s. Any chat window where nobody types for a minute, any realtime dashboard between server pushes, any keepalive-less protocol: they all get their sockets clubbed at exactly the one-minute mark. Raising the timeout to one hour is aggressive but reasonable for most WebSocket apps, and it lines up with common load balancer ceilings. If your app has an application-level keepalive that fires every 30 seconds you can leave this lower, but only if you trust that keepalive to be present in every client build.

Knob three, proxy_send_timeout 3600s. Same idea but for the other direction, bytes going from nginx to the upstream. Chat backends that stream events out but rarely receive them can hit this timeout if you leave it at the default, especially during a network partition when the client TCP window shrinks to zero and nginx can't flush its send buffer.

Knob four, keepalive_timeout 65s on the server block, matched by keepalive 64 on the upstream. The upstream pool means nginx doesn't open a fresh TCP connection to Bun on every handshake, which cuts handshake latency from a full three-way TCP handshake plus TLS to a single message on a warm socket. On a benchmark run against localhost, handshake p95 dropped from 12 ms to 3 ms after this change alone. The 65s on the server block is deliberately larger than a common upstream load balancer default of 60s, so the LB is always the party that closes the socket first. Symmetric timeouts on both sides of a proxy hop cause TCP resets and confused logs, so pick a small offset and stick with it.

Step 5: prove it with 200 sockets and ten minutes (commit 6ebf2d4)

Anecdotes are cheap. A load test isn't. The stress/soak.js k6 script opens 200 WebSocket connections against nginx on port 8080, sends one small ping every 90 seconds per socket (deliberately longer than the default 60s timeout so the "sockets die at exactly 60s" bug is unmissable), and holds them for ten minutes:

export const options = {
  scenarios: {
    soak: { executor: "constant-vus", vus: 200, duration: "10m" },
  },
};

export default function () {
  ws.connect(WS_URL, {}, function (socket) {
    socket.on("open", () => {
      socket.setInterval(() =>
        socket.send(JSON.stringify({ ts: Date.now() })), 90_000);
    });
    socket.setTimeout(() => socket.close(), 600_000);
  });
}

The interesting comparison is running the same script against three different checkouts of the same repo. Numbers below are from a MacBook Pro M2 running Docker Desktop 4.30 with default resource caps.

Checkout	Handshake success	Sockets alive at 5 min	Sockets alive at 10 min	Handshake p95
Step 2 (adf4d2b)	0 / 200	0 / 200	0 / 200	n/a
Step 3 (91d9db7)	200 / 200	0 / 200	0 / 200	12 ms
Step 4 (4eb7053)	200 / 200	200 / 200	200 / 200	3 ms

The Step 3 column is the one that surprises people. Everything looks fine for a minute. The Grafana graph is green. Then the whole thing tips over as proxy_read_timeout fires on every socket at once. Real production incidents look exactly like this: a deploy at 14:00, no errors for a minute, then a simultaneous 100% reconnection storm at 14:01 as the client library tries to redial the whole connection pool. If your alerts are wired only to error rate and not to WebSocket connection count, you never see the incident begin.

Repository

Full source at https://github.com/vytharion/nginx-websocket-reverse-proxy.

• Step 0 init: 62fc2b4. Project scaffold with .gitignore and README.
• Step 1: 71f2d1d. Bun WebSocket echo server on port 3000.
• Step 2: adf4d2b. Naive nginx proxy that silently drops the Upgrade handshake.
• Step 3: 91d9db7. proxy_set_header Upgrade and Connection via map.
• Step 4: 4eb7053. proxy_buffering off, upstream keepalive pool, one hour idle timeouts.
• Step 5: 6ebf2d4. k6 soak script that holds 200 sockets for 10 minutes.

Clone it, docker compose up --build, then git checkout each SHA in turn and watch the behaviour change one knob at a time.

Next steps

There are three places to take this once the four knobs work.

TLS termination. Move the port 443 listener into nginx with a Let's Encrypt cert, keep the port 80 listener as a redirect to wss://, and put proxy_pass http://ws_upstream untouched behind the TLS listener. nginx will terminate TLS itself and the Upgrade and Connection header dance is identical. The one thing to remember is that browsers refuse wss:// from a page loaded over https:// if the cert isn't valid, so test end to end from a real domain rather than localhost with --insecure toggled on.

Sticky sessions. If you scale the upstream to more than one Bun instance behind the same nginx and your app stores per-socket state in memory, you'll need ip_hash (or hash $remote_addr consistent) on the upstream block so a reconnecting client lands on the same instance. Most modern WebSocket backends persist state to Redis specifically to sidestep this, and if you can it's the easier path.

Rate limiting the handshake. The four knobs above keep long-lived sockets alive but do nothing about a client that reconnects a thousand times a second. Add limit_req on location /ws with a burst that matches your expected reconnection storm, and either the client backs off gracefully or it gets 503, which is the correct outcome for a rogue caller. The limit_req_zone reference at https://nginx.org/en/docs/http/ngx_http_limit_req_module.html walks through the two-line configuration.

The repo is small enough to fork, and every step is a real commit. Break it, patch it, run the k6 script, watch the numbers move. That's how the four knobs stop being magic and start being the two-line diff you already know how to write next time this bug shows up in production.