The Request-Response Bottleneck: Short Polling, Long Polling & The Latency-Overhead Tradeoff

Real-time is not a library you import, but an architectural contract across the network stack: choose the lowest-complexity transport that matches your data symmetry, budget for long-lived socket file descriptors, and decouple stateful client connections from stateless event producers via a distributed backplane. This series builds that contract from first principles.

Every real-time system begins with the same naive pattern: ask the server if anything changed. The client polls. The server responds. Repeat every second. This is polling — and most teams condemn it without ever quantifying why, leaving them to reach for WebSockets in contexts where a CDN-cached conditional HTTP request would be orders of magnitude cheaper, more reliable, and easier to operate.

This article derives the polling overhead equation from wire-level first principles, implements event-driven long polling that resolves via in-memory bridges rather than database spin-locks, and maps both transport choices to their CAP theorem alignment. By the end, you will have a precise, quantitative framework for deciding when polling is the correct architectural choice — and exactly where it breaks.

1. The HTTP Request-Response Baseline

1.1 The TCP 3-Way Handshake and TLS 1.3 Overhead

Before a single byte of application data is transmitted, every new HTTP connection pays a mandatory infrastructure tax. Understanding this tax is the foundation of every real-time transport tradeoff.

A cold HTTP/1.1 request with a new TCP connection:

Three round trips on a cold connection. At 50ms RTT (typical US cross-region latency), that is 150ms of pure infrastructure overhead before the first application byte arrives. TLS 1.3 reduces this from the previous 4-RTT TLS 1.2 handshake down to 2 RTTs for the session establishment phase, and even supports 0-RTT resumption for previously seen clients — but only if the session ticket is valid and the application code is safe against replay attacks.

HTTP Keep-Alive changes the calculus. Once a TCP connection is established, HTTP/1.1 reuses it for subsequent requests via the Connection: keep-alive header. A warm polling loop only pays:

1 RTT per poll request (application data round trip)
+ 0 RTT TCP setup (connection already open)
+ 0 RTT TLS setup (session already established)

This is the steady-state cost that matters for polling calculations.

1.2 The Packet Overhead Equation

Before debating polling interval trade-offs, let's calculate the actual on-wire overhead for a typical polling request against a real endpoint.

A minimal short-poll HTTP request to /api/vehicles/positions:

GET /api/vehicles/positions?lastUpdate=1729792800000 HTTP/1.1
Host: api.fleettracker.io
Authorization: Bearer eyJhbGciOiJIUzI1NiJ9...
Cookie: session=abc123def456...
Accept: application/json
Accept-Encoding: gzip, deflate, br
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (compatible; FleetTracker/1.0)

Measured header size: ~520–780 bytes of HTTP request headers, depending on cookie length and token size.

Typical application payload: A { "hasUpdates": false } response is approximately 22 bytes of JSON. A list of 50 vehicle positions might be 4KB.

The overhead ratio for a "nothing changed" poll response:

Overhead Ratio = Request Headers / (Request Headers + Payload)
               = 650 bytes / (650 + 22 bytes) ≈ 97%

For every 100 bytes transmitted in a high-frequency polling loop returning empty responses, only 3 bytes are application data. The remaining 97% is protocol scaffolding.

2. Short Polling: Mechanics, Latency Bounds & Edge Caching

2.1 The Expected Latency Equation

Short polling works by sending requests at a fixed interval I. If an event occurs at a uniformly random time within that interval, the expected delivery latency is:

$$E[\text{latency}] = \frac{I}{2}$$

Where:

• I: The polling interval in seconds/milliseconds.
• E[latency]: The average time between an event occurring on the server and the client observing it.

At a 1-second polling interval, clients observe events with an average 500ms delay. At 5 seconds, it is 2.5 seconds. The worst-case delivery latency is always the full interval I.

The server load compound effect. If you have C concurrent clients polling at interval I, the server receives C / I requests per second. Doubling the polling frequency (halving I) doubles the request rate. Doubling the number of clients doubles it again. Combined:

$$\text{RPS} = \frac{C}{I}$$

2.2 Conditional Requests & CDN Edge Caching

For cacheable, low-velocity data (public vehicle positions updated every 5 seconds, sports scores, stock summary snapshots), HTTP conditional requests transform short polling from a bandwidth problem into a highly efficient edge-served pattern.

The server implementation:

// ❌ Naive poll — hits database every time, returns full payload
app.get('/api/vehicles/summary', async (req, res) => {
  const data = await db.vehicles.findAll() // DB query on every request
  return res.json(data)
})

// ✅ Conditional ETag poll — CDN serves 304 Not Modified for unchanged data
app.get('/api/vehicles/summary', async (req, res) => {
  const data = await db.vehicles.findAll()
  const etag = `"${computeHash(data)}"` // stable hash of payload content

  // ETag validation — matches current resource fingerprint
  if (req.headers['if-none-match'] === etag) {
    return res.status(304).end() // 22 bytes — CDN intercepts before origin
  }

  res
    .set('ETag', etag)
    .set('Cache-Control', 'public, max-age=5, stale-while-revalidate=2')
    .json(data)
})

The client polling loop with conditional validation:

// ✅ Client-side conditional polling — zero payload when data unchanged
async function pollVehicleSummary(interval = 5000): Promise<void> {
  let lastEtag: string | null = null

  while (true) {
    try {
      const headers: HeadersInit = { Accept: 'application/json' }
      if (lastEtag) headers['If-None-Match'] = lastEtag

      const res = await fetch('/api/vehicles/summary', { headers })

      if (res.status === 304) {
        // Nothing changed — CDN responded, origin never touched
        // bandwidth cost: ~22 bytes for the 304 response line
      } else if (res.ok) {
        lastEtag = res.headers.get('ETag')
        const data = await res.json()
        renderVehicleMap(data) // update UI only when data actually changed
      }
    } catch (err) {
      console.warn('Poll failed, retrying:', err)
    }

    await sleep(interval)
  }
}

With a CDN in front (Cloudflare, AWS CloudFront), the 304 Not Modified response is served entirely from the edge — the origin Express server is never invoked until the ETag changes. This transforms polling from a per-client database hit into a single origin revalidation every 5 seconds, regardless of how many thousands of clients are polling.

2.3 When Short Polling is the Correct Choice

3. Long Polling: Event-Driven Hanging GETs

3.1 The Core Mechanism

Long polling — also called the "Comet" pattern or "hanging GET" — eliminates the delivery latency of short polling by keeping the HTTP request open on the server until either an event fires or a timeout is reached.

The request lifecycle:

1. Client sends GET /api/events/subscribe.
2. Server receives the request but does not respond immediately. It holds the res object and registers a listener.
3. When an event occurs (vehicle position update, order status change), the server writes the response and closes the request.
4. Client receives the response, processes the event, and immediately re-opens a new hanging request.
5. If no event fires within the timeout window (typically 25–30 seconds), the server sends an empty { "events": [] } heartbeat response.

3.2 The Anti-Pattern: Database Spin-Lock Inside a Hanging Request

The most common long polling implementation mistake is replacing the short polling interval with a polling loop inside the hanging request:

// ❌ Database spin-lock inside a hanging GET — wrong in every way
app.get('/api/events/subscribe', async (req, res) => {
  const since = Number(req.query.since) || Date.now()
  const timeout = Date.now() + 25_000

  // Spin-polling the database from inside the held request
  while (Date.now() < timeout) {
    const events = await db.events.find({ createdAt: { $gt: since } })
    if (events.length > 0) return res.json({ events })
    await sleep(500) // 500ms database poll interval
  }

  return res.json({ events: [] })
})

This is a disaster:

• It does not eliminate database queries — it just moves them server-side. 1,000 concurrent clients still generate 2 database queries per second per client (2,000 total QPS) during the spin loop.
• Node.js is single-threaded and event-loop driven. The await sleep(500) releases the event loop between iterations, but the accumulation of 1,000 simultaneous database queries will saturate the database connection pool and stall the event loop.
• It provides no delivery latency improvement over short polling — events are still only observed at the next 500ms spin iteration.

3.3 The Correct Pattern: In-Memory EventEmitter Bridge

The correct architecture decouples the hanging request from any polling mechanism entirely. An in-memory EventEmitter (or Redis Pub/Sub channel in a multi-node setup) serves as the bridge:

// server/lib/EventBridge.ts
import { EventEmitter } from 'node:events'

// Module-level singleton — lives in Node.js process memory
const bridge = new EventEmitter()
bridge.setMaxListeners(0) // Unlimited listeners (each connected client = 1 listener)

export function publishEvent(tenantId: string, event: FleetEvent): void {
  bridge.emit(`tenant:${tenantId}`, event)
}

export function subscribeOnce(
  tenantId: string,
  onEvent: (event: FleetEvent) => void,
  onTimeout: () => void,
  timeoutMs: number = 25_000,
): () => void {
  const channel = `tenant:${tenantId}`

  const handler = (event: FleetEvent) => {
    clearTimeout(timer)
    bridge.removeListener(channel, handler)
    onEvent(event)
  }

  const timer = setTimeout(() => {
    bridge.removeListener(channel, handler)
    onTimeout()
  }, timeoutMs)

  bridge.once(channel, handler)

  // Return cleanup function for req.on('close') handler
  return () => {
    clearTimeout(timer)
    bridge.removeListener(channel, handler)
  }
}

// server/routes/events.ts
import { publishEvent, subscribeOnce } from '../lib/EventBridge'

// ✅ Event-driven long polling — zero database queries, zero spin loops
app.get('/api/events/subscribe', requireAuth, (req, res) => {
  const tenantId = req.user!.tenantId

  // Critical: Set timeout longer than the long poll window
  req.socket.setTimeout(0) // Disable socket timeout for this connection

  // Clean up if client disconnects early (tab close, navigation, network drop)
  let cleanup: (() => void) | null = null

  req.on('close', () => {
    cleanup?.()
  })

  cleanup = subscribeOnce(
    tenantId,
    // Event fires: resolve immediately
    (event) => {
      res
        .status(200)
        .json({ events: [event], timestamp: Date.now() })
    },
    // Timeout: send heartbeat, client will immediately reconnect
    () => {
      res
        .status(200)
        .json({ events: [], timestamp: Date.now() })
    },
  )
})

// Your existing API routes write events and publish to the bridge
app.post('/api/vehicles/:id/position', requireAuth, async (req, res) => {
  const event = await db.vehicles.updatePosition(
    req.params.id,
    req.body
  )

  // Publish to in-memory bridge — zero-cost if no one is listening
  publishEvent(req.user!.tenantId, event)

  res.status(200).json({ success: true })
})

What makes this correct:

• The database is queried only once per write — when the event is created. Zero queries during the hanging wait period.
• Each waiting client is a single EventEmitter listener — memory cost is approximately 120 bytes per listener, not a database connection.
• Event delivery latency is sub-millisecond — the EventEmitter.emit() call is synchronous within the Node.js event loop.
• The req.on('close') handler prevents ghost listeners when clients disconnect abruptly.

Expand

3.4 Client-Side Reconnection Loop

The client must immediately re-open the hanging request after each response:

// ✅ Production long-poll client with proper reconnection
class LongPollClient {
  private abortController: AbortController | null = null
  private running = false

  async start(tenantId: string, onEvents: (events: FleetEvent[]) => void) {
    this.running = true
    this.poll(tenantId, onEvents)
  }

  stop() {
    this.running = false
    this.abortController?.abort()
  }

  private async poll(tenantId: string, onEvents: (events: FleetEvent[]) => void) {
    while (this.running) {
      this.abortController = new AbortController()

      try {
        const res = await fetch('/api/events/subscribe', {
          signal: this.abortController.signal,
          headers: { Authorization: `Bearer ${getToken()}` },
        })

        if (!res.ok) throw new Error(`Poll failed: ${res.status}`)

        const { events } = await res.json() as { events: FleetEvent[] }

        if (events.length > 0) {
          onEvents(events)
        }
        // Empty heartbeat? Just continue the loop immediately
        // No sleep needed — server already enforced the 25s wait window

      } catch (err) {
        if ((err as Error).name === 'AbortError') break // Intentional stop

        // Network error — back off before retry
        await sleep(Math.min(30_000, 1_000 * Math.pow(2, this.retryCount++)))
      }
    }
  }

  private retryCount = 0
}

4. The Browser 6-Connection Pool Deadlock

4.1 HTTP/1.1 Connection Limits

HTTP/1.1 browsers enforce a hard limit of 6 simultaneous TCP connections per origin (per scheme + host + port combination). This limit exists to prevent any single website from monopolizing network resources.

When a long-polling or SSE connection holds one of those 6 slots, it occupies it for the entire 25-second timeout window. Open multiple tabs to the same application:

Browser Tab 1 -> Long Poll connection (slot 1 occupied: 25s hold)
Browser Tab 2 -> Long Poll connection (slot 2 occupied: 25s hold)
Browser Tab 3 -> Long Poll connection (slot 3 occupied: 25s hold)
Browser Tab 4 -> Long Poll connection (slot 4 occupied: 25s hold)
Browser Tab 5 -> Long Poll connection (slot 5 occupied: 25s hold)
Browser Tab 6 -> Long Poll connection (slot 6 occupied: 25s hold)

Browser Tab 1 -> clicks a link -> GET /api/data
  => ALL 6 SLOTS EXHAUSTED -- request queued indefinitely

Every subsequent REST API call from any tab is blocked until a poll response frees a slot. The application becomes unresponsive.

HTTP/2 resolves this entirely. HTTP/2 multiplexes all requests over a single TCP connection using independent stream IDs. A single HTTP/2 connection to api.fleettracker.io can carry 100+ simultaneous SSE streams, long-poll requests, and REST calls concurrently without any inter-stream blocking.

5. Proxy & Gateway Timeouts: The Silent Connection Killer

5.1 Nginx Default Timeouts

When long-polling requests pass through a reverse proxy, the proxy enforces its own read timeout — the maximum duration it will wait for a response from the upstream server before returning an error to the client.

Nginx default: proxy_read_timeout 60s;

If your long-poll timeout window is 60 seconds or longer, Nginx silently terminates the connection with a 504 Gateway Timeout before the server's heartbeat fires. The client receives a network error rather than an empty heartbeat, triggering unintended error-path reconnect logic.

# /etc/nginx/sites-available/fleettracker.conf

server {
    location /api/events/subscribe {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;

        # Must be longer than server-side poll timeout window
        proxy_read_timeout 35s; # Server timeout is 25s → safe margin

        # Prevent Nginx from buffering the response
        proxy_buffering off;

        # Keep the upstream connection alive
        proxy_set_header Connection '';
    }
}

The heartbeat is your safety valve. The server's periodic empty response (every 20–25 seconds) serves dual purposes:

1. Proxy keep-alive: Resets the proxy's idle read timer before it fires.
2. Client liveness signal: The client confirms the connection is still healthy and immediately re-opens the poll.

5.2 The Connection Storm (Thundering Herd)

The most dangerous failure mode in long polling is what happens when a broadcast event fires and thousands of waiting clients all receive responses simultaneously:

The mitigation is client-side jitter. After receiving a response, clients must not reconnect immediately with zero delay — they must apply a small randomized jitter window:

// ❌ Instant reconnect — creates thundering herd
await pollOnce()
poll() // reconnects immediately

// ✅ Jittered reconnect — disperses reconnection spike
await pollOnce()
await sleep(Math.random() * 500) // 0–500ms random jitter
poll() // reconnected connections distributed over 500ms window

With 5,000 clients reconnecting uniformly distributed across 500ms, the reconnection arrival rate is 10,000 req/s rather than 5,000 req/s in a 10ms spike — a 50x reduction in instantaneous server pressure.

Expand

6. Resource Profiling: Polling at Scale

6.1 Memory & File Descriptor Cost

Every open HTTP connection in Node.js consumes an OS file descriptor. The OS imposes a hard limit on file descriptors per process (ulimit -n). Default Linux limits are 1,024 or 4,096 depending on the distribution. This must be raised before deploying any long-running connection server:

# Check current limit
ulimit -n  # Default: 1024 on many systems

# Set per-process limit (add to /etc/security/limits.conf for persistence)
ulimit -n 65536

# For Node.js systemd services:
# /etc/systemd/system/fleettracker.service
[Service]
LimitNOFILE=65536

Memory cost comparison per connection type:

6.2 When to Reach for Something Else

Long polling starts failing at scale under specific conditions:

7. CAP Theorem Alignment

7.1 Short Polling: AP Semantics

Short polling is the purest Availability + Partition Tolerance (AP) transport pattern in the CAP theorem sense:

• Availability: Every poll request gets a deterministic response (even if stale). There is no blocking wait for a quorum or leader node. Any read replica can serve the response.
• Partition Tolerance: When a network partition isolates a database replica, short poll clients continue receiving responses — they receive the last known state rather than an error.
• Consistency: Sacrificed. The delivery latency bound E[latency] = I/2 represents a bounded staleness window. Clients will always observe data that is at most I seconds old.

This makes short polling the correct choice for AP-preferring distributed reads: public leaderboards, cached feeds, and read-heavy dashboards where temporary staleness is acceptable.

7.2 Long Polling: AP with Ephemeral State

Long polling introduces ephemeral server-side state (the hanging request and its associated EventEmitter listener). Under a network partition:

• If the client is partitioned from the server, the hanging request is dropped at the TCP level. The client detects this via a connection error and re-polls from its last known state.
• The server's in-memory listener is garbage-collected when req.on('close') fires.
• No data is permanently lost — the client simply misses events that occurred during the partition window and re-syncs on reconnect.

Long polling is AP with bounded gaps: it provides low-latency delivery under normal conditions but degrades gracefully to short-polling semantics during partitions, with the client picking up from its last known checkpoint.

Summary

What's Next

Long polling solves the delivery latency problem of short polling, but holds one HTTP/1.1 connection per client and cannot serve bidirectional data push efficiently at scale. In Part 2, we examine Server-Sent Events (SSE) — a W3C standard that provides a persistent, unidirectional server-to-client HTTP stream with native browser auto-reconnection, Last-Event-ID state reconciliation, and HTTP/2 multiplexing across dozens of independent event channels over a single TCP connection.

Part 2 → Server-Sent Events: High-Throughput Unidirectional Streaming over HTTP — realtime-server-sent-events-sse-streaming (coming in Part 2 of this series)