Siddhant Deval
Siddhant Deval
backend22 min read

Kafka Streams: Stateful Processing, Windows, and KTable

Kafka Streams turns Kafka from a transport layer into a compute layer — stateful aggregations, windowed joins, and KTable materializations run inside your application process with no external stream-processing cluster required. This article covers KStream vs KTable, all four window types, stream-table joins, and exactly-once processing guarantees in Kafka Streams.

Kafka Streams: Stateful Processing, Windows, and KTable

The product team wants a live dashboard showing orders per minute for each region, updated in real time. The first implementation polls a PostgreSQL aggregation query every 5 seconds: SELECT region, COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '1 minute' GROUP BY region. At 10,000 orders per minute, this query runs a full table scan every 5 seconds, generating measurable read load on the primary. The team adds a Redis counter, updated by the order service on every insert. Now they have a cache invalidation problem: when an order is cancelled, the counter must be decremented — but the order service and the analytics cache have no transactional relationship.

Kafka Streams solves both problems without a separate stream-processing cluster: the aggregation runs inside the application process, state is maintained in local RocksDB backed by a compacted Kafka changelog topic, and the result is re-published to a Kafka topic that any consumer can read.

Architectural Note

Series positioning: This is Part 4 of Messaging at Cloud Scale. Kafka Streams is the compute layer that sits on top of the transport layer covered in Series 1. The prerequisite mental models are Event Sourcing (append-log semantics) and CQRS (read model projections from write-model events). This article uses the Kafka Streams Java DSL — the same patterns apply to ksqlDB for SQL-based stream processing.


1. KStream vs KTable: The Fundamental Distinction

KStream: an unbounded sequence of events — each record is an independent fact
KTable:  a materialized view — each record is the latest value for a key (upsert semantics)
JAVA
// KStream vs KTable mental model
StreamsBuilder builder = new StreamsBuilder();

// KStream: every message is an independent event
// orders topic: [ {orderId: 1, region: "EU", total: 99}, {orderId: 2, region: "US", total: 149}, ... ]
KStream<String, Order> orderStream = builder.stream("orders.created");
// All order events, including duplicates, cancellations, partial events

// KTable: latest value per key (compacted topic — only the latest value per key is kept)
// users topic: [ {userId: "u1", tier: "free"}, {userId: "u1", tier: "premium"}, ... ]
// KTable sees: { "u1": { tier: "premium" } }  — only the latest
KTable<String, User> userTable = builder.table("users",
    Consumed.with(Serdes.String(), userSerde));
// userTable.toStream() emits only when a key changes — changelog semantics
Crucial Requirement

KTable is not a cache. Every KTable update is durably written to a compacted changelog topic in Kafka. If the Streams process restarts, it restores KTable state from the changelog before resuming processing — no data loss, no warm-up period. A cache reset on restart; a KTable does not.


2. Stateful Aggregations

2.1 Count Aggregation per Window

JAVA
// ✅ Orders per minute per region — tumbling window aggregation
KStream<String, Order> orders = builder.stream("orders.created",
    Consumed.with(Serdes.String(), orderSerde));

KTable<Windowed<String>, Long> ordersPerMinutePerRegion = orders
    // Rekey by region — region becomes the aggregation key
    .selectKey((orderId, order) -> order.region())
    // Tumbling window: non-overlapping, fixed-duration buckets
    .groupByKey(Grouped.with(Serdes.String(), orderSerde))
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)))
    .count(Materialized.as("orders-per-minute-store"));

// Emit results to a topic — each window result is a Kafka record
ordersPerMinutePerRegion.toStream()
    .map((windowedKey, count) -> KeyValue.pair(
        windowedKey.key(),   // region
        new OrderCount(windowedKey.key(), windowedKey.window().start(), count)
    ))
    .to("orders.metrics.per-minute", Produced.with(Serdes.String(), orderCountSerde));

2.2 The Four Window Types

Window Type Definition Use case
Tumbling Fixed size, no overlap — [0:00–1:00], [1:00–2:00], ... Orders per minute, hourly totals
Hopping Fixed size, overlapping — [0:00–1:00], [0:30–1:30], ... Rolling averages, moving sums
Session Variable size, closed by inactivity gap User session duration, activity bursts
Sliding Emits when a new record falls in/out of window boundary Real-time anomaly detection
JAVA
// Tumbling: fixed buckets, no overlap
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)))

// Hopping: 1-minute window, advancing every 30 seconds
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).advanceBy(Duration.ofSeconds(30)))

// Session: close window after 30 minutes of inactivity
.windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30)))
Pro Tip & Optimization

Use session windows for user activity tracking — they model real behaviour better than tumbling windows. A user who spends 3 minutes on a page, leaves for 45 minutes, and returns creates two sessions under session windows (gap = 30 min), but one inflated tumbling bucket if you use hourly windows. Session windows produce variable-length windows that reflect actual inactivity — the correct abstraction for behavioural analytics.


3. KTable and Stream-Table Joins

3.1 Enriching Events with KTable Lookups

JAVA
// ✅ Stream-table join: enrich every order event with current user tier
KStream<String, Order> orders = builder.stream("orders.created",
    Consumed.with(Serdes.String(), orderSerde));

KTable<String, User> users = builder.table("users",
    Consumed.with(Serdes.String(), userSerde));

// Rekey orders by userId to match the KTable key
KStream<String, Order> ordersByUser = orders
    .selectKey((orderId, order) -> order.customerId());

// Join: for each order, look up the user's current tier from the KTable
KStream<String, EnrichedOrder> enriched = ordersByUser.join(
    users,
    (order, user) -> new EnrichedOrder(order, user.tier(), user.loyaltyPoints()),
    // KTable join: always uses the CURRENT value of the KTable key
    // If user.tier changes mid-stream: new orders get the new tier, old orders are unaffected
    Joined.with(Serdes.String(), orderSerde, userSerde)
);

enriched.to("orders.enriched", Produced.with(Serdes.String(), enrichedOrderSerde));

3.2 KTable Materialized State Store

JAVA
// ✅ Interactive queries: expose KTable state as a queryable REST endpoint
// Kafka Streams makes the RocksDB state store queryable from within the application

ReadOnlyKeyValueStore<String, Long> orderCountStore = streams.store(
    StoreQueryParameters.fromNameAndType(
        "orders-per-minute-store",
        QueryableStoreTypes.keyValueStore()
    )
);

// Query the materialized store directly — no Kafka round-trip
long count = orderCountStore.get("EU");  // current EU order count from local RocksDB
// Returns in < 1ms from local store — zero network I/O

4. Exactly-Once Processing Guarantee

JAVA
// ✅ Exactly-once within Kafka Streams topology
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,          "order-aggregator");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,       "kafka:9092");
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
    StreamsConfig.EXACTLY_ONCE_V2);  // requires Kafka 2.5+
// Effect: input consumption + state update + output production are atomic per task
// A process crash mid-processing: Kafka Streams replays from last committed checkpoint
// Result: each input record produces exactly one output record, state updated exactly once
Performance / Safety Warning

processing.guarantee=exactly_once_v2 provides exactly-once semantics within the Kafka Streams topology — from input topic to state store to output topic. It does not extend exactly-once to external sinks. If your Streams topology writes enriched events to a PostgreSQL database via a Kafka Connect sink, the DB write is at-least-once. External sinks still require idempotency keys (ON CONFLICT DO NOTHING) for end-to-end exactly-once business semantics.


Summary

Concept Rule
KTable is a materialized view KTable is not a cache — it is a materialized view of a compacted Kafka topic; every update is durably stored in the changelog and survives process restart.
Exactly-once scope processing.guarantee=exactly_once_v2 scopes transactions to the Kafka Streams task boundary — it does not extend exactly-once to external sinks; DB writes still need idempotency keys.
Session vs Tumbling windows Session windows are the correct abstraction for user activity sessions with variable inactivity gaps; tumbling windows are for fixed-interval metrics (e.g., orders per minute).

What's Next

Part 5: Saga Choreography and Event-Driven Workflow Patterns addresses the distributed transaction problem that appears when a business operation spans multiple services — each with its own database, its own failure modes, and no shared transaction coordinator. Saga choreography, compensating transactions, and the process manager pattern keep multi-service workflows consistent without two-phase commit.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Kafka Streams#KTable#Stream Processing#Windowing#ksqlDB#Stateful Processing#Backend
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.