Siddhant Deval
Siddhant Deval
system design5 min read

Full-Stack AI Observability & Tracing with OpenTelemetry

Debugging distributed AI architectures with console.log is impossible when a request spans React, Node.js middleware, PostgreSQL vector queries, and upstream LLM streams. This article instruments the full Workspace stack with OpenTelemetry GenAI semantic conventions, providing end-to-end distributed traces, token attribution, and latency waterfall debugging across all seven tiers.

Series·Part 23 of 24

Full-Stack AI Architecture: From Streaming Foundations to Resilient Agentic Systems in Node.js & React

Full-Stack AI Observability & Tracing with OpenTelemetry

"The AI is slow" is not an actionable incident report. Is it slow because the HNSW query is taking 400ms? Because the cross-encoder reranker is timing out? Because the LLM provider's P99 spiked? Because the React client is queuing SSE events faster than it can render? Without distributed tracing that spans from the React component through the Fastify gateway, pgvector retrieval, and the streaming LLM call, all of these look identical from the outside: a slow response.

This article instruments the complete Workspace stack with OpenTelemetry following the GenAI semantic conventions: TTFT and ITL as span attributes on LLM calls, per-tenant token attribution for cost analysis, retrieval spans with chunk counts and reranker scores, and a React TTFT measurement hook that feeds the first-token timestamp into a browser span exported to the same trace.

Architectural Note

Who this is for: Engineers debugging AI system performance with only server access logs and no visibility into which layer of a multi-stage pipeline is the bottleneck. After this article, a single Grafana trace view shows the exact latency breakdown across every layer.


1. GenAI Semantic Conventions

The OpenTelemetry GenAI Semantic Conventions (spec v0.9+) define standard attribute names for LLM spans, enabling vendor-neutral instrumentation that any observability backend understands.

Attribute Value Example
gen_ai.provider.name 'anthropic', 'openai'
gen_ai.operation.name 'chat', 'text_completion', 'embeddings'
gen_ai.request.model 'claude-3-5-sonnet-20241022'
gen_ai.request.max_tokens 4096
gen_ai.response.model 'claude-3-5-sonnet-20241022'
gen_ai.response.finish_reason 'end_turn', 'max_tokens', 'stop'
gen_ai.usage.input_tokens 1284
gen_ai.usage.output_tokens 823
gen_ai.workspace.ttft_ms 1340 (custom)
gen_ai.workspace.org_id 'org-uuid' (custom)

2. OpenTelemetry Setup

TYPESCRIPT
// ✅ OTel SDK initialisation — must run before any other imports
// file: src/telemetry/otel.ts

import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'
import { Resource } from '@opentelemetry/resources'
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions'
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-node'

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'workspace-ai-gateway',
    [SemanticResourceAttributes.SERVICE_VERSION]: process.env.APP_VERSION ?? '0.0.0',
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV,
  }),
  spanProcessors: [
    new SimpleSpanProcessor(
      new OTLPTraceExporter({
        url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4317',
      }),
    ),
  ],
})

sdk.start()
process.on('SIGTERM', () => sdk.shutdown())

export { sdk }

3. LLM Call Instrumentation

TYPESCRIPT
// ✅ Instrumented LLM streaming call with GenAI semantic conventions
// file: src/telemetry/InstrumentedLlmCall.ts

import { trace, SpanStatusCode, SpanKind } from '@opentelemetry/api'

const tracer = trace.getTracer('workspace-ai-gateway', '1.0.0')

export async function* instrumentedStreamCall(
  orgId: string,
  model: string,
  inputTokens: number,
  callFn: () => AsyncGenerator<WorkspaceAction>,
): AsyncGenerator<WorkspaceAction> {
  const span = tracer.startSpan('gen_ai.chat', {
    kind: SpanKind.CLIENT,
    attributes: {
      // gen_ai.provider.name replaces deprecated gen_ai.system (OTel GenAI conventions v0.9+)
      'gen_ai.provider.name': model.startsWith('claude') ? 'anthropic' : 'openai',
      'gen_ai.operation.name': 'chat',
      'gen_ai.request.model': model,
      'gen_ai.usage.input_tokens': inputTokens,
      'gen_ai.workspace.org_id': orgId,
    },
  })

  const requestSentAt = performance.now()
  let firstTokenAt: number | null = null
  let outputTokenCount = 0
  let prevTokenAt = requestSentAt
  const itlSamples: number[] = []

  try {
    for await (const action of callFn()) {
      const now = performance.now()

      if (firstTokenAt === null) {
        firstTokenAt = now
        span.setAttribute('gen_ai.workspace.ttft_ms', Math.round(now - requestSentAt))
      } else {
        itlSamples.push(now - prevTokenAt)
      }

      prevTokenAt = now
      outputTokenCount++
      yield action
    }

    // Emit final metrics as span attributes
    span.setAttributes({
      'gen_ai.usage.output_tokens': outputTokenCount,
      'gen_ai.workspace.p50_itl_ms': percentile(itlSamples, 50),
      'gen_ai.workspace.p99_itl_ms': percentile(itlSamples, 99),
      'gen_ai.workspace.total_ms': Math.round(performance.now() - requestSentAt),
      'gen_ai.response.finish_reason': 'end_turn',
    })

    span.setStatus({ code: SpanStatusCode.OK })
  } catch (error) {
    span.recordException(error as Error)
    span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message })
    throw error
  } finally {
    span.end()
  }
}

function percentile(values: number[], p: number): number {
  if (values.length === 0) return 0
  const sorted = [...values].sort((a, b) => a - b)
  const idx = Math.floor((p / 100) * sorted.length)
  return Math.round(sorted[Math.min(idx, sorted.length - 1)])
}

4. RAG Pipeline Span

TYPESCRIPT
// ✅ Retrieval pipeline span — wraps hybrid search + reranking
// file: src/telemetry/InstrumentedRetrieval.ts

import { trace, SpanStatusCode, SpanKind } from '@opentelemetry/api'

const tracer = trace.getTracer('workspace-ai-gateway')

export async function instrumentedRetrieve(
  orgId: string,
  query: string,
  retrieveFn: () => Promise<RankedResult[]>,
): Promise<RankedResult[]> {
  return tracer.startActiveSpan('workspace.rag.retrieve', {
    kind: SpanKind.INTERNAL,
    attributes: {
      'workspace.retrieval.org_id': orgId,
      'workspace.retrieval.query_length': query.length,
    },
  }, async (span) => {
    const hybridStart = performance.now()

    try {
      const results = await retrieveFn()

      span.setAttributes({
        'workspace.retrieval.hybrid_search_ms': Math.round(performance.now() - hybridStart),
        'workspace.retrieval.chunks_returned': results.length,
        'workspace.retrieval.top_score': results[0]?.rerankerScore?.toFixed(3) ?? '0',
        'workspace.retrieval.total_tokens': results.reduce((s, r) => s + r.tokenCount, 0),
      })

      span.setStatus({ code: SpanStatusCode.OK })
      return results
    } catch (error) {
      span.recordException(error as Error)
      span.setStatus({ code: SpanStatusCode.ERROR })
      throw error
    } finally {
      span.end()
    }
  })
}

5. Browser TTFT Measurement (React)

TYPESCRIPT
// ✅ Client-side TTFT span — exported to OTel collector via OTLP/HTTP
// file: apps/web/src/telemetry/ClientTtftSpan.ts

import { WebTracerProvider } from '@opentelemetry/sdk-trace-web'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { trace } from '@opentelemetry/api'

const provider = new WebTracerProvider()
provider.addSpanProcessor(
  new SimpleSpanProcessor(
    new OTLPTraceExporter({ url: '/api/otel/traces' })
  )
)
provider.register()

const webTracer = trace.getTracer('workspace-web-client', '1.0.0')

export function measureClientTtft(
  parentTraceId: string,  // From SSE response header X-Trace-Id
): { markFirstToken: () => void } {
  const requestSentAt = performance.now()

  const span = webTracer.startSpan('client.sse.ttft', {
    attributes: {
      'gen_ai.workspace.server_trace_id': parentTraceId,
      'gen_ai.workspace.client_ua': navigator.userAgent.slice(0, 100),
    },
  })

  return {
    markFirstToken: () => {
      span.setAttribute('gen_ai.workspace.client_ttft_ms', Math.round(performance.now() - requestSentAt))
      span.end()
    },
  }
}

Summary

Layer Span Key Attributes
React Client client.sse.ttft client_ttft_ms, server_trace_id
Fastify Route http.server (auto) http.status_code, http.route
RAG Retrieval workspace.rag.retrieve chunks_returned, top_score, hybrid_search_ms
LLM Call gen_ai.chat ttft_ms, p99_itl_ms, input_tokens, output_tokens

What's Next

The final part of this series — Part 24 — assembles every component from Parts 1–23 into the complete Enterprise Collaborative AI Workspace production deployment: end-to-end system diagram, team conventions, performance benchmarks, and the architecture decision record for each key design choice.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#OpenTelemetry#Observability#Tracing#Node.js#React#LLM#Monitoring#GenAI
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.

Series·Part 23 of 24

Full-Stack AI Architecture: From Streaming Foundations to Resilient Agentic Systems in Node.js & React

Curriculum Roadmap
01The Full-Stack AI Paradigm Shift: Beyond Toy Wrappers & Prompt Concatenation
02The Anatomy of an LLM Call: Tokens, Context Windows & Probabilistic Mechanics
03Structured Outputs & Formal Invariants: Taming Non-Determinism with Zod & TypeBox
04Full-Stack Streaming Protocols: SSE, Fetch Streams & WebSockets
05V8 Streams, Server-Side Backpressure & Cancellations in Node.js
06React 18/19 Streaming UI & Generative Components
07Generative UI & Interactive Canvas Synchronization
08High-Dimensional Vector Embeddings & Similarity Math
09Vector Storage & Indexing at Scale: HNSW vs IVF in PostgreSQL (pgvector)
10Hybrid Search: Dense Vectors Meet Sparse Lexical BM25
11Multi-Stage Retrieval & Cross-Encoder Reranking
12Advanced RAG Architectures: Contextual Retrieval & Parent-Document Splitting
13Query Transformation, Decomposition & Multi-Query Expansion
14Dynamic Context Window Management & Long-Context Packing
15Agent Loops & Deterministic State Machines
16Model Context Protocol (MCP) in Full-Stack Architecture
17Human-in-the-Loop & Durable Agent Workflows
18Multi-Agent Orchestration & Collaborative Swarms
19Semantic Caching & Provider Cost Optimization
20Multi-Provider AI Gateways: Fallbacks, Rate Limiting & Hedged Requests
21Client-Side AI & Edge Hybrid Execution: WebGPU & Transformers.js
22Production Evals & Guardrails: Measuring Retrieval and Generation
23Full-Stack AI Observability & Tracing with OpenTelemetry
Reading
24Production Capstone: Assembling The Enterprise Collaborative AI Workspace