WebSockets Under the Hood: RFC 6455 Framing, the Upgrade Handshake & Raw ws vs socket.io

Real-time is not a library you import, but an architectural contract across the network stack: choose the lowest-complexity transport that matches your data symmetry, budget for long-lived socket file descriptors, and decouple stateful client connections from stateless event producers via a distributed backplane. Parts 1 and 2 established that polling and SSE serve unidirectional or infrequent data push over ordinary HTTP. The moment your architecture requires genuine bidirectionality — the client sends commands, the server sends state, and both channels operate concurrently at sub-50ms latency — HTTP's request-response model cannot deliver it. WebSockets can.

But "just use Socket.IO" is not an architectural decision — it is an abdication of one. Socket.IO bundles Engine.IO, a custom transport abstraction, packet framing, namespace multiplexing, and an automatic polling fallback. Each of these choices carries memory overhead, network cost, and scaling constraints that are invisible until you hit production scale. This article builds the complete mental model from the TCP wire up: the 101 handshake, the binary frame layout, the 4-byte masking key requirement, OS socket resource math, and a precise benchmark-driven comparison between raw ws and Socket.IO.

1. The HTTP 101 Upgrade Handshake

1.1 Why WebSockets Start as HTTP

WebSockets hijack an existing HTTP/1.1 request to establish a persistent TCP connection. This design decision was intentional: it allows WebSocket traffic to traverse existing HTTP infrastructure (firewalls, proxies, load balancers) that expect HTTP on port 80/443 without requiring new port allocations.

The handshake uses standard HTTP headers to negotiate the protocol switch:

Client initiates:

GET /ws/fleet HTTP/1.1
Host: api.fleettracker.io
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Origin: https://fleettracker.io

Server responds:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

After the 101 response, the TCP connection is permanently taken over by the WebSocket protocol. HTTP is done. No more HTTP verbs, no more headers, no more request-response cycles — raw binary frames flow in both directions over the same socket.

1.2 The Sec-WebSocket-Accept Validation

The Sec-WebSocket-Accept header is a cryptographic challenge-response that proves the server actually understands the WebSocket protocol (preventing non-WebSocket HTTP servers from accidentally accepting upgrade requests):

The magic GUID (258EAFA5-E914-47DA-95CA-C5AB0DC85B11) is a fixed string specified in RFC 6455. It is hardcoded — never changes. Its purpose is to prevent a non-WebSocket-aware HTTP server from accidentally accepting a WebSocket upgrade by echoing back whatever the client sent.

// The exact computation the server performs — verifiable in any language
import { createHash } from 'node:crypto'

function computeWebSocketAccept(clientKey: string): string {
  const MAGIC_GUID = '258EAFA5-E914-47DA-95CA-C5AB0DC85B11'
  return createHash('sha1')
    .update(clientKey + MAGIC_GUID)
    .digest('base64')
}

// computeWebSocketAccept('dGhlIHNhbXBsZSBub25jZQ==')
// → 's3pPLMBiTxaQ9kYGzzhZRbK+xOo='

2. Dissecting the RFC 6455 Wire Frame

2.1 The Binary Frame Header Layout

Once the handshake completes, all communication uses a compact binary frame format. Understanding the frame structure is essential for debugging network issues, calculating overhead, and evaluating serialization choices.

WebSocket Frame Layout (RFC 6455 §5.2):

Bit: 0 1 2 3 4 5 6 7 | 8 9 A B C D E F | 16..23 | 24..31
     -----------------------------------------------------
     F R R R | Opcode  | M | Payload Len   | (Extended length if needed)
     I S S S |  (4b)   | A |    (7b)       |
     N V V V |         | S |               |
       1 2 3 |         | K |               |

Minimum frame overhead:

• Small payload (≤125 bytes): 2 bytes of header (no masking key for server frames)
• Client frame with masking key: 6 bytes of header (2 header + 4 masking key bytes)
• Compare to HTTP/1.1: 520–800 bytes of headers

Expand

2.2 Why Client-to-Server Frames MUST Be Masked

The 4-byte XOR masking requirement for client-to-server frames is not encryption — a fixed key in the same frame provides zero security. It is a poison pill against cache-poisoning attacks on transparent HTTP proxies.

The attack scenario without masking:

A malicious attacker crafts a WebSocket payload that looks like a valid HTTP response:

GET /admin HTTP/1.1\r\n
Host: target.example.com\r\n
\r\n
HTTP/1.1 200 OK\r\n
Content-Length: 0\r\n
\r\n

An intermediate transparent proxy that does not understand WebSockets but intercepts the traffic might interpret this binary stream as a legitimate HTTP exchange and cache the fake response, poisoning its cache for other users on the same network.

XOR masking with a random 4-byte key makes it statistically impossible to craft a payload that looks like a valid HTTP request/response after masking — because the mask key changes randomly for every frame.

// The XOR masking algorithm (applied byte-by-byte)
function maskPayload(payload: Buffer, maskKey: Buffer): Buffer {
  const masked = Buffer.allocUnsafe(payload.length)
  for (let i = 0; i < payload.length; i++) {
    masked[i] = payload[i]! ^ maskKey[i % 4]! // XOR with cycling mask key bytes
  }
  return masked
}

// Unmasking is identical — XOR is its own inverse
const unmasked = maskPayload(maskedPayload, maskKey)

2.3 OpCodes and Control Frames

Control frames (Close, Ping, Pong) must have FIN=1 and must never exceed 125 bytes of payload. They can be interleaved between data frame fragments.

3. OS & Node.js Resource Reality

3.1 File Descriptors

Every open TCP socket — including WebSocket connections — consumes one OS file descriptor. The OS kernel maintains a table of open file descriptors per process, with a hard ceiling enforced by ulimit.

# Current per-process file descriptor limit
ulimit -n        # Typical default: 1024 (macOS) or 4096 (Linux)

# Current system-wide file descriptor limit
cat /proc/sys/fs/file-max   # e.g., 9,223,372,036,854,775,807

# Set for current session
ulimit -n 65536

# Persistent: /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536

# Systemd service override
[Service]
LimitNOFILE=65536

At the default ulimit -n 1024, your Node.js WebSocket server can accept at most ~1,000 concurrent connections before the OS returns EMFILE: too many open files and rejects new socket accept() calls. Every new Node.js WebSocket server must raise this limit before deployment.

3.2 Socket Buffer Memory

Beyond file descriptors, each TCP socket allocates kernel memory for send and receive buffers. These are separate from Node.js heap memory — they live in kernel space:

# Current TCP socket buffer sizes
cat /proc/sys/net/core/rmem_default    # Default receive buffer: 212992 bytes (~208KB)
cat /proc/sys/net/core/wmem_default    # Default send buffer: 212992 bytes (~208KB)
cat /proc/sys/net/core/rmem_max        # Max receive buffer: 134217728 bytes (128MB)

# The actual per-socket memory allocation:
# - Minimum: 4KB (low-traffic idle socket)
# - Default: 208KB send + 208KB receive ≈ 416KB per socket
# - Under load (backpressure): may grow to rmem_max

Memory calculation for a WebSocket server:

At 50,000 concurrent connections, a single Node.js WebSocket server needs approximately 800MB of combined kernel + heap memory under idle conditions. Active connections with full send buffers can easily require 3–4× more.

3.3 Application-Level Ping/Pong: Why TCP Keep-Alive Is Insufficient

TCP has its own keep-alive mechanism — but it is tuned for infrastructure failure detection, not application health monitoring:

# Default TCP keep-alive values on Linux
cat /proc/sys/net/ipv4/tcp_keepalive_time     # 7200 seconds (2 hours!)
cat /proc/sys/net/ipv4/tcp_keepalive_intvl    # 75 seconds
cat /proc/sys/net/ipv4/tcp_keepalive_probes   # 9 probes

TCP keep-alive takes up to 2 hours + 675 seconds (9 × 75s) to detect a dead socket at default settings. A half-open socket — where one end is dead but the other doesn't know — can persist for hours, consuming a file descriptor and memory buffer the entire time.

Application-level WebSocket Ping/Pong is the only reliable liveness mechanism:

// server/lib/WebSocketServer.ts
import { WebSocketServer, WebSocket } from 'ws'

const wss = new WebSocketServer({ port: 3001 })

const PING_INTERVAL = 30_000  // 30 seconds
const PONG_TIMEOUT  = 10_000  // 10 seconds to respond

wss.on('connection', (socket: WebSocket) => {
  let isAlive = true
  let pingTimer: ReturnType<typeof setTimeout>

  socket.on('pong', () => {
    // Client responded to our ping — connection is alive
    isAlive = true
  })

  function schedulePing() {
    pingTimer = setTimeout(() => {
      if (!isAlive) {
        // No pong received within timeout — half-open socket, terminate
        console.log('Dead socket detected — terminating')
        socket.terminate() // Force TCP RST — no graceful close frame
        return
      }

      isAlive = false // Will be reset to true when pong arrives
      socket.ping()   // RFC 6455 OpCode 0x9
      schedulePing()  // Schedule next ping
    }, PING_INTERVAL)
  }

  schedulePing()

  socket.on('close', () => clearTimeout(pingTimer))
})

4. Raw ws vs socket.io — The Architectural Boundary

4.1 What Is Engine.IO?

socket.io is not a WebSocket server. It is a messaging framework layered on Engine.IO, which is itself a transport abstraction layer. Engine.IO's behavior on connection:

The consequence: Socket.IO always initiates connections via HTTP long polling before attempting a WebSocket upgrade. This requires:

1. Multiple HTTP requests to establish a single Socket.IO connection.
2. The initial polling requests have a sid (session ID) in the query string.
3. If you load-balance across multiple Node.js instances without sticky sessions, the upgrade POST and the polling GET may land on different instances — which have no knowledge of each other's sid. The result: 400 Session ID Unknown errors.

4.2 Packet Overhead Comparison

Raw ws frame (UTF-8 JSON text, 48 bytes of payload):

[0x81] [0x30]                    = 2 bytes header
{"type":"position","lat":37.77}  = 48 bytes payload
---------------------------------
Total: 50 bytes (4.0% overhead)

Socket.IO packet wrapping the same payload:

[0x81] [0xB9]                              = 2 bytes WebSocket header
[0x34] [0x32]                              = 2 bytes Engine.IO packet type ("42" = message)
["fleet:position",{"lat":37.77,...}]       = 54 bytes Socket.IO event envelope
-----------------------------------------
Total: 58 bytes (16.7% overhead over payload, 16% larger than raw ws)

Expand

The overhead is small in absolute terms. At 10,000 messages per second, Socket.IO sends ~80KB/s more data than raw ws for the same events. The more significant overhead is CPU — Socket.IO must parse the Engine.IO packet type, strip the namespace prefix, deserialize the event name from the array envelope, and route to the correct event handler.

4.3 Server Memory Benchmark: 10,000 Concurrent Sockets

// Benchmark harness: 10,000 concurrent idle connections + 1 msg/sec/client
// Hardware: 8-core, 16GB RAM, Node.js 22, Linux 6.x

// Results (measured via process.memoryUsage() and /proc/PID/status):
const benchmarkResults = {
  rawWs: {
    heapUsedMB: 312,        // V8 heap for socket objects + message buffers
    rssMB: 480,             // Resident Set Size (heap + native bindings + OS stack)
    msgLatencyP95: '0.8ms', // 95th percentile message delivery latency
    msgLatencyP99: '2.1ms',
    throughputMsgSec: 10_000,
  },
  socketIo: {
    heapUsedMB: 541,        // +73% vs raw ws
    rssMB: 720,             // +50% RSS due to Engine.IO state per socket
    msgLatencyP95: '2.3ms', // Engine.IO packet parsing overhead
    msgLatencyP99: '6.8ms',
    throughputMsgSec: 10_000,
  },
}

4.4 When Socket.IO Is the Correct Choice

4.5 A Minimal Production ws Server

// server/ws/FleetWebSocketServer.ts
import { createServer } from 'node:https'
import { WebSocketServer, WebSocket } from 'ws'
import { ConnectionManager } from './ConnectionManager'

const httpsServer = createServer({ /* TLS options */ })
const wss = new WebSocketServer({ server: httpsServer, path: '/ws/fleet' })
const manager = new ConnectionManager()

wss.on('connection', (socket: WebSocket, req) => {
  // Authenticate from the HTTP upgrade request headers
  const ticket = new URL(req.url!, 'http://localhost').searchParams.get('ticket')
  const session = validateTicket(ticket)

  if (!session) {
    socket.close(1008, 'Unauthorized') // Policy Violation
    return
  }

  const connId = manager.register(socket, session)

  socket.on('message', (data, isBinary) => {
    if (isBinary) {
      // Handle MessagePack / binary frame
      handleBinaryMessage(connId, data as Buffer)
    } else {
      const msg = JSON.parse(data.toString()) as ClientMessage
      handleMessage(connId, msg)
    }
  })

  socket.on('close', (code, reason) => {
    manager.unregister(connId)
    console.log(`Connection ${connId} closed: ${code} ${reason}`)
  })

  socket.on('error', (err) => {
    console.error(`Socket error on ${connId}:`, err)
    manager.unregister(connId)
  })
})

5. Serialization: JSON vs Binary Formats

For high-throughput telemetry (vehicle positions, sensor readings), JSON serialization is often the bottleneck — not network bandwidth:

// Serialization benchmark: 1,000,000 encode/decode cycles

// JSON (built-in)
// Encode: ~380ms  Decode: ~290ms  Size: 87 bytes for typical vehicle payload

// MessagePack (msgpackr)
// Encode: ~82ms   Decode: ~76ms   Size: 52 bytes (40% smaller than JSON)
// npm install msgpackr

import { pack, unpack } from 'msgpackr'

// Server: encode binary
const binaryPayload = pack({
  v: 'B12',        // vehicleId
  t: Date.now(),   // timestamp
  la: 37.7749,     // lat
  ln: -122.4194,   // lng
  s: 67,           // speed
})
socket.send(binaryPayload) // Sends as WebSocket Binary frame (opcode 0x2)

// Client: decode binary
socket.binaryType = 'arraybuffer'
socket.addEventListener('message', (e) => {
  if (e.data instanceof ArrayBuffer) {
    const vehicle = unpack(new Uint8Array(e.data))
    updateMap(vehicle)
  }
})

6. CAP Alignment: Network Partitions & Half-Open Sockets

6.1 Silent TCP Partitions

In the CAP theorem, a network partition is a communication failure that prevents some nodes from reaching others. For WebSocket servers, the most dangerous partition is the half-open socket: the client's OS has experienced a network interruption, but the server has not yet detected it (TCP keep-alive has not fired yet, which can take 2 hours at default settings).

During this partition:

• The server holds the file descriptor, kernel buffers, and all application-level state (ConnectionManager entries, room memberships).
• Messages sent by the server are queued in the kernel send buffer, waiting for TCP acknowledgment that never arrives.
• From the server's perspective, the client appears fully connected.

This is the availability side of the CAP trade-off for WebSockets: the server remains available (accepting new connections, processing messages from other clients), but the consistency of the "connected" state is violated — the server believes the client is present when it is not.

Application-level Ping/Pong (Section 3.3) is the only reliable mechanism to detect half-open sockets quickly. It converts a potential 2-hour detection window into a 40-second window (30s ping interval + 10s pong timeout).

Summary

What's Next

We now have a raw WebSocket server that handles the RFC 6455 handshake, manages Ping/Pong liveness detection, and maintains a ConnectionManager. But managing stateful connections inside an Express/Node.js application introduces integration challenges that most tutorials skip: React 18/19 StrictMode double-mount leaks, secure authentication on the WebSocket handshake (not in query strings), and the architectural coupling between database change detection and socket distribution. Part 4 solves these with a complete stateful connection management system including cryptographic ticket authentication, a singleton ConnectionManager, and a Single Ingestion Worker pattern over MongoDB Replica Set Change Streams.

Part 4 → Stateful Connection Management: Resilient Client Lifecycles & Database Change Streams — realtime-stateful-connection-management-change-streams (coming in Part 4 of this series)