Siddhant Deval
Siddhant Deval
backend5 min read

The Full-Stack AI Paradigm Shift: Beyond Toy Wrappers & Prompt Concatenation

Most engineers treat LLMs like deterministic CRUD endpoints, wrapping raw fetch calls in ad-hoc controller functions that collapse under real concurrency. This article establishes the probabilistic microservice mental model and defines the decoupled, contract-driven architecture of the Enterprise Collaborative AI Workspace that runs throughout the series.

Series·Part 1 of 24

Full-Stack AI Architecture: From Streaming Foundations to Resilient Agentic Systems in Node.js & React

The Full-Stack AI Paradigm Shift: Beyond Toy Wrappers & Prompt Concatenation

Treat LLMs as non-deterministic probabilistic microservices: isolate their outputs behind strict invariant contracts, govern streams with deterministic backpressure, and bridge intent to state through typed tools, durable loops, and verifiable retrieval. That mindset runs through every article in this series. Before we build the streaming gateway, the vector knowledge vault, or the agent worker mesh, we need to be precise about why the approach most engineers take first — wrapping a raw fetch() call inside a controller function — fails under real production conditions.

This article defines the architectural vocabulary, establishes the boundary model that governs the entire series, and introduces the Enterprise Collaborative AI Workspace: the unified domain every subsequent article extends.

Architectural Note

Who this is for: Node.js and React engineers with 4+ years of production experience who have integrated LLMs into applications and run into unexplained failures, spiralling costs, or brittle systems that break when prompts change. No prior ML or data science background is required.


1. The Broken Pattern: LLM-as-a-Deterministic-Endpoint

1.1 How Engineers Reach for LLMs First

The instinct is understandable. An OpenAI or Anthropic HTTP call looks structurally identical to any other third-party REST API call: send a request, receive a response, handle errors, move on. So engineers reach for the same patterns.

TYPESCRIPT
// ❌ Anti-Pattern — LLM treated as a deterministic REST endpoint
// file: src/controllers/AiController.ts

import express from 'express'
import OpenAI from 'openai'

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })
const router = express.Router()

router.post('/chat', async (req, res) => {
  const { message, history } = req.body

  // Raw string concatenation — no schema, no validation, no budget
  const prompt = history.map((m: any) => `${m.role}: ${m.content}`).join('\n')
  
  try {
    const completion = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [
        { role: 'system', content: 'You are a helpful assistant.' },
        ...history,
        { role: 'user', content: message },
      ],
    })

    // Treating model output as trusted data — zero validation
    const aiResponse = completion.choices[0].message.content

    // Directly passing model output to a downstream mutation
    await db.messages.create({ role: 'assistant', content: aiResponse })

    res.json({ reply: aiResponse })
  } catch (error) {
    res.status(500).json({ error: 'AI failed' }) // Opaque, unrecoverable
  }
})

This code ships to production and works fine on the demo dataset. Three weeks later, production reports start arriving:

  • The endpoint occasionally takes 22 seconds and then returns a 504 from the upstream proxy — model inference spiked under load.
  • A conversation with 40 turns caused a silent HTTP 400: the context window overflowed, and the error was swallowed.
  • A user's message about their "SQL injection attack surface audit" triggered a false-positive safety filter, returning an empty response with no indication to the client.
  • Two concurrent requests for the same user created a race condition in db.messages.create, producing duplicate assistant messages.
  • After a prompt change deployed to staging, the model started returning responses that failed JSON.parse() — silently breaking a downstream feature that depended on structured output.

Each of these failures stems from the same root cause: the model is non-deterministic and the architecture treats it as if it were not.

1.2 What Makes an LLM Non-Deterministic at the Infrastructure Level

Non-determinism in LLMs is not just about temperature settings or creative variance in outputs. It is an infrastructure-level property that affects every layer of a full-stack system:

Failure Dimension Deterministic API Assumption LLM Reality
Latency P99 < 200ms, stable variance TTFT: 300ms–5s; Total: 2s–45s depending on output length
Output Shape Defined by schema/contract Probabilistic tokens; schema adherence requires enforcement
Error Surface HTTP 4xx/5xx, typed exceptions Safety filters, context overflow, rate limits, quota exhaustion
Idempotency Same input → same output Same input → different token sequence each call
Throughput Scales linearly with horizontal pods Rate-limited per-organisation; TPM/RPM hard ceilings
Cost Fixed per-request or per-unit Variable per input+output token; context length directly multiplies cost
Performance / Safety Warning

The most expensive production AI bugs are not model quality issues — they are architectural mismatches. Engineers debug prompt content for hours while the actual failure is an unhandled context overflow, an uncancelled stream burning tokens for disconnected clients, or a race condition in state mutation after a non-idempotent model call.


2. The Correct Mental Model: Probabilistic Microservice Boundary

2.1 Treating the LLM as an External Non-Deterministic Service

The correct abstraction is to treat the LLM as a third-party non-deterministic microservice that sits outside your trust boundary — the same way a mature system treats a payment processor or an external inventory API. You do not trust its output shape. You do not assume its latency. You do not couple your domain state mutations directly to its responses.

The gateway is the trust boundary. Every token that comes out of the LLM passes through it. Nothing from the LLM touches the database, the client state, or the canvas without first being validated, schema-checked, and authorised.

2.2 The Four Invariants of a Production AI Gateway

Before any AI-generated data can cross the trust boundary into domain state, it must satisfy all four invariants:

Invariant Enforcement Mechanism Violation Consequence
Structural Validity JSON Schema + Zod strict parse Reject payload; trigger AST repair or re-prompt
Token Budget Context Window Scheduler (Part 14) Truncate history; prune RAG chunks; never overflow
Cancellation Safety AbortController wired to req.on('close') Stop upstream generation; prevent billing leak
Rate Compliance Token-Bucket per-org, per-user, per-model 429 fast-fail; failover to backup provider
TYPESCRIPT
// ✅ Correct Pattern — Gateway with trust boundary
// file: src/gateway/AiGateway.ts

import { z } from 'zod'
import { StreamingGateway } from './StreamingGateway'
import { ContextScheduler } from './ContextScheduler'
import { RateLimiter } from './RateLimiter'

// Every AI response shape is declared as a Zod schema at the boundary
const WorkspaceActionSchema = z.object({
  type: z.enum(['canvas_node', 'text_reply', 'tool_call']),
  payload: z.unknown(),
  confidence: z.number().min(0).max(1),
}).strict()

type WorkspaceAction = z.infer<typeof WorkspaceActionSchema>

export class AiGateway {
  constructor(
    private readonly streaming: StreamingGateway,
    private readonly scheduler: ContextScheduler,
    private readonly limiter: RateLimiter,
  ) {}

  async *streamResponse(
    request: GatewayRequest,
    signal: AbortSignal,
  ): AsyncGenerator<WorkspaceAction> {
    // Invariant 1: Rate compliance before touching the provider
    await this.limiter.acquire(request.orgId, request.estimatedTokens)

    // Invariant 2: Context scheduled before sending to model
    const scheduledMessages = this.scheduler.pack(request.messages, request.ragChunks)

    // Stream from provider — tokens stay raw until validated
    for await (const rawChunk of this.streaming.stream(scheduledMessages, signal)) {
      // Invariant 3: Structural validity at the boundary
      const parsed = WorkspaceActionSchema.safeParse(rawChunk)
      if (!parsed.success) {
        // Invariant violation: log, repair, or skip — never crash
        continue
      }
      yield parsed.data // Only validated, typed data exits the gateway
    }
  }
}
Crucial Requirement

Notice that AiGateway is completely decoupled from Express, Fastify, or any HTTP framework. It accepts an AbortSignal and yields typed domain objects. The transport layer (SSE, WebSocket, gRPC) is a concern of the caller, not the gateway — this is the dependency inversion that makes the gateway testable without mocking HTTP.


3. The Enterprise Collaborative AI Workspace

3.1 The Domain We Are Building

Every article in this series is a chapter in building the same system: the Enterprise Collaborative AI Workspace. This is a multi-tenant SaaS application that provides:

  • Streaming AI Chat: A multi-turn conversation interface where responses stream token-by-token, tool invocations render as interactive UI cards, and the context window is dynamically managed.
  • Collaborative Generative Canvas: A real-time shared diagram and document canvas where AI agents and human users edit nodes simultaneously, with conflict resolution.
  • Hybrid Vector Knowledge Vault: A PostgreSQL + pgvector knowledge base that engineers can upload documents to. The AI retrieves relevant context using hybrid dense+sparse search with reranking.
  • Background MCP Agent Workers: Autonomous agents running on durable execution runtimes that perform research, draft documents, and propose code changes — pausing for human approval before any destructive action.

3.2 What Each Tier of This Series Builds

Tier Parts What Gets Built in the Workspace
Foundations 1–3 Gateway architecture, token budgeting harness, Zod schema validation pipeline
Streaming Backbone 4–7 SSE streaming pipeline, Node.js backpressure gateway, React 60fps streaming renderer, live canvas widget streamer
Vector Foundations 8–11 AST-aware document chunker, HNSW pgvector knowledge vault, Hybrid RRF search SQL, Cross-encoder reranker
Production RAG 12–14 Contextual Retrieval ingestion pipeline, Query transformation router, Context Window Scheduler
Agents & MCP 15–18 FSM Agent Runtime, Custom MCP Server + OAuth 2.1, Durable Human-in-the-Loop worker, Multi-agent supervisor
Performance 19–21 Redis Semantic Cache, Multi-provider gateway with hedged requests, WebGPU on-device PII filter
Production Ops 22–24 Ragas CI eval pipeline, OpenTelemetry GenAI tracing, Full production capstone assembly

3.3 The Technology Stack (Fixed for the Entire Series)

All subsequent articles write code against these exact versions. There are no "you could also use X" digressions — the stack is chosen and justified here.

Layer Technology Justification
Server Runtime Node.js 22 LTS (ESM) Native Web Streams (ReadableStream, TransformStream) without polyfills; AbortSignal.any() for composed cancellation
Language TypeScript 5.4+ Stage 3 Decorators; using keyword for deterministic resource cleanup
HTTP Framework Fastify 5 Zero-overhead serialisation; native async hooks for request lifecycle; plugin architecture for streaming
Database PostgreSQL 16 + pgvector 0.7+ HNSW indexing; Row-Level Security; single-engine for relational + vector data; ACID for canvas mutations
Cache Redis 7+ Pub/Sub for real-time canvas sync; Vector search module for semantic caching
ORM Drizzle ORM Raw SQL escape hatches for pgvector operators (<=>) and RANK() OVER window functions
Frontend React 19 + TypeScript useSyncExternalStore for streaming message stores; Actions for server-driven mutations
AI SDK Vercel AI SDK Core (ai) Provider-agnostic streaming abstractions; Structured Object generation
Agent Protocol MCP TypeScript SDK v2 @modelcontextprotocol/server + @modelcontextprotocol/client; Streamable HTTP transport
In-Browser AI Transformers.js v3 + WebGPU ONNX Runtime for SLM inference; WebGPU compute shaders
Pro Tip & Optimization

The choice of Drizzle over Prisma is deliberate. Prisma's query engine abstracts away the raw SQL required for pgvector <=> distance operators and RANK() OVER window functions needed for Reciprocal Rank Fusion (RRF) in Part 10. Drizzle exposes typed raw SQL via sql\`` template literals that preserve TypeScript inference without escaping to untyped strings.


4. Local Development Environment

4.1 Setting Up Reproducible Local Infrastructure

Before connecting to any paid cloud model, every component must run locally with deterministic mock providers. This is not optional infrastructure — it is the prerequisite for test-driven AI development.

BASH
# docker-compose.yml — local infrastructure stack
YAML
services:
  postgres:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_DB: workspace_dev
      POSTGRES_USER: dev
      POSTGRES_PASSWORD: dev_secret
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis/redis-stack:7.4.0-v0
    ports:
      - "6379:6379"  # Redis
      - "8001:8001"  # RedisInsight UI

  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    # Run after start: docker exec ollama ollama pull llama3.2:3b
    # Run after start: docker exec ollama ollama pull nomic-embed-text

volumes:
  postgres_data:
  ollama_data:
TYPESCRIPT
// ✅ Correct Pattern — Provider-agnostic model client factory
// file: src/providers/ModelClientFactory.ts

import { createOpenAI } from '@ai-sdk/openai'
import { createAnthropic } from '@ai-sdk/anthropic'

export type ModelProvider = 'anthropic' | 'openai' | 'ollama'

export function createModelClient(provider: ModelProvider) {
  switch (provider) {
    case 'anthropic':
      return createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY! })
    case 'openai':
      return createOpenAI({ apiKey: process.env.OPENAI_API_KEY! })
    case 'ollama':
      // Local Ollama — zero cost, deterministic for tests
      return createOpenAI({
        baseURL: 'http://localhost:11434/v1',
        apiKey: 'ollama', // required by SDK but unused
      })
  }
}

// Environment-driven selection: tests always use 'ollama'
export const defaultProvider: ModelProvider =
  (process.env.AI_PROVIDER as ModelProvider) ?? 'ollama'

4.2 Project Structure for the Workspace

enterprise-ai-workspace/
├── apps/
│   ├── web/                      ← React 19 frontend (Vite)
│   │   ├── src/
│   │   │   ├── features/
│   │   │   │   ├── chat/         ← Streaming chat UI (Parts 6, 7)
│   │   │   │   ├── canvas/       ← Collaborative canvas (Part 7, 18)
│   │   │   │   └── approvals/    ← Human-in-the-loop UI (Part 17)
│   │   │   └── workers/
│   │   │       └── pii.worker.ts ← WebGPU PII filter (Part 21)
│   └── gateway/                  ← Fastify Node.js AI Gateway
│       ├── src/
│       │   ├── gateway/          ← Trust boundary (Parts 1–5)
│       │   ├── retrieval/        ← RAG pipeline (Parts 8–14)
│       │   ├── agents/           ← FSM + MCP workers (Parts 15–18)
│       │   └── cache/            ← Semantic caching (Part 19)
├── packages/
│   ├── db/                       ← Drizzle schema (PostgreSQL + pgvector)
│   ├── ai-schemas/               ← Shared Zod schemas (Part 3)
│   └── mcp-tools/                ← Custom MCP server (Part 16)
└── docker-compose.yml
Crucial Requirement

The apps/gateway/src/gateway/ directory is the trust boundary defined in §2. Nothing outside this directory communicates directly with an AI provider. All model calls go through AiGateway. This constraint is enforced by ESLint import rules in Part 22 CI gates.


5. The Failure Modes You Will Stop Making

Before moving to Part 2, be precise about what the architecture in this series eliminates. These are real production failures with measurable cost:

Failure Mode Cause Series Article That Fixes It
V8 heap exhaustion under concurrent streams Piping model stream directly to HTTP response, no backpressure Part 5
Orphaned generation loops billing after disconnects No req.on('close') → AbortController.abort() chain Part 5
React 60fps re-render storm on token arrival Appending tokens directly to useState Part 6
Context window silent overflow → empty response No token budget scheduler Part 14
Vector search blind spots on exact identifiers Pure dense embedding search, no BM25 hybrid Part 10
Isolated chunk ambiguity in RAG Chunks embedded without parent document context Part 12
Unbounded agent loops burning tokens while(true) tool-calling without step budget Part 15
Vendor lock-in + 429 cascade Single hardcoded provider, no circuit breaker Part 20
Prompt regression in production No automated CI eval gate on AI component changes Part 22

Summary

Concept Rule
Probabilistic Microservice Treat LLMs as non-deterministic external services — never trust their output shape, latency, or idempotency.
Trust Boundary Every AI-generated token must pass through a schema gate before touching domain state or client UI.
Four Invariants Structural Validity, Token Budget, Cancellation Safety, and Rate Compliance must all hold before any model output is accepted.
Provider Abstraction The gateway decouples domain logic from model providers; switching providers is a configuration change, not a code change.
Local First All development starts with local Ollama models and containerised infrastructure before connecting to paid cloud providers.

What's Next

In Part 2, we open the black box of a single LLM API call. We dissect BPE tokenisation mechanics, the pre-fill vs. decode phase split, KV cache reuse, and why TTFT and ITL are the latency metrics that actually determine streaming UI responsiveness — not total request duration.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#AI Architecture#Node.js#TypeScript#LLM#System Design#Full-Stack
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.

Series·Part 1 of 24

Full-Stack AI Architecture: From Streaming Foundations to Resilient Agentic Systems in Node.js & React

Curriculum Roadmap
01The Full-Stack AI Paradigm Shift: Beyond Toy Wrappers & Prompt Concatenation
Reading
02The Anatomy of an LLM Call: Tokens, Context Windows & Probabilistic Mechanics
03Structured Outputs & Formal Invariants: Taming Non-Determinism with Zod & TypeBox
04Full-Stack Streaming Protocols: SSE, Fetch Streams & WebSockets
05V8 Streams, Server-Side Backpressure & Cancellations in Node.js
06React 18/19 Streaming UI & Generative Components
07Generative UI & Interactive Canvas Synchronization
08High-Dimensional Vector Embeddings & Similarity Math
09Vector Storage & Indexing at Scale: HNSW vs IVF in PostgreSQL (pgvector)
10Hybrid Search: Dense Vectors Meet Sparse Lexical BM25
11Multi-Stage Retrieval & Cross-Encoder Reranking
12Advanced RAG Architectures: Contextual Retrieval & Parent-Document Splitting
13Query Transformation, Decomposition & Multi-Query Expansion
14Dynamic Context Window Management & Long-Context Packing
15Agent Loops & Deterministic State Machines
16Model Context Protocol (MCP) in Full-Stack Architecture
17Human-in-the-Loop & Durable Agent Workflows
18Multi-Agent Orchestration & Collaborative Swarms
19Semantic Caching & Provider Cost Optimization
20Multi-Provider AI Gateways: Fallbacks, Rate Limiting & Hedged Requests
21Client-Side AI & Edge Hybrid Execution: WebGPU & Transformers.js
22Production Evals & Guardrails: Measuring Retrieval and Generation
23Full-Stack AI Observability & Tracing with OpenTelemetry
24Production Capstone: Assembling The Enterprise Collaborative AI Workspace