Kafka Streams: Stateful Processing, Windows, and KTable
Kafka Streams turns Kafka from a transport layer into a compute layer — stateful aggregations, windowed joins, and KTable materializations run inside your application process with no external stream-processing cluster required. This article covers KStream vs KTable, all four window types, stream-table joins, and exactly-once processing guarantees in Kafka Streams.
Messaging at Cloud Scale
Kafka Streams: Stateful Processing, Windows, and KTable
The product team wants a live dashboard showing orders per minute for each region, updated in real time. The first implementation polls a PostgreSQL aggregation query every 5 seconds: SELECT region, COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '1 minute' GROUP BY region. At 10,000 orders per minute, this query runs a full table scan every 5 seconds, generating measurable read load on the primary. The team adds a Redis counter, updated by the order service on every insert. Now they have a cache invalidation problem: when an order is cancelled, the counter must be decremented — but the order service and the analytics cache have no transactional relationship.
Kafka Streams solves both problems without a separate stream-processing cluster: the aggregation runs inside the application process, state is maintained in local RocksDB backed by a compacted Kafka changelog topic, and the result is re-published to a Kafka topic that any consumer can read.
Series positioning: This is Part 4 of Messaging at Cloud Scale. Kafka Streams is the compute layer that sits on top of the transport layer covered in Series 1. The prerequisite mental models are Event Sourcing (append-log semantics) and CQRS (read model projections from write-model events). This article uses the Kafka Streams Java DSL — the same patterns apply to ksqlDB for SQL-based stream processing.
1. KStream vs KTable: The Fundamental Distinction
KTable is not a cache. Every KTable update is durably written to a compacted changelog topic in Kafka. If the Streams process restarts, it restores KTable state from the changelog before resuming processing — no data loss, no warm-up period. A cache reset on restart; a KTable does not.
2. Stateful Aggregations
2.1 Count Aggregation per Window
2.2 The Four Window Types
| Window Type | Definition | Use case |
|---|---|---|
| Tumbling | Fixed size, no overlap — [0:00–1:00], [1:00–2:00], ... |
Orders per minute, hourly totals |
| Hopping | Fixed size, overlapping — [0:00–1:00], [0:30–1:30], ... |
Rolling averages, moving sums |
| Session | Variable size, closed by inactivity gap | User session duration, activity bursts |
| Sliding | Emits when a new record falls in/out of window boundary | Real-time anomaly detection |
Use session windows for user activity tracking — they model real behaviour better than tumbling windows. A user who spends 3 minutes on a page, leaves for 45 minutes, and returns creates two sessions under session windows (gap = 30 min), but one inflated tumbling bucket if you use hourly windows. Session windows produce variable-length windows that reflect actual inactivity — the correct abstraction for behavioural analytics.
3. KTable and Stream-Table Joins
3.1 Enriching Events with KTable Lookups
3.2 KTable Materialized State Store
4. Exactly-Once Processing Guarantee
processing.guarantee=exactly_once_v2 provides exactly-once semantics within the Kafka Streams topology — from input topic to state store to output topic. It does not extend exactly-once to external sinks. If your Streams topology writes enriched events to a PostgreSQL database via a Kafka Connect sink, the DB write is at-least-once. External sinks still require idempotency keys (ON CONFLICT DO NOTHING) for end-to-end exactly-once business semantics.
Summary
| Concept | Rule |
|---|---|
| KTable is a materialized view | KTable is not a cache — it is a materialized view of a compacted Kafka topic; every update is durably stored in the changelog and survives process restart. |
| Exactly-once scope | processing.guarantee=exactly_once_v2 scopes transactions to the Kafka Streams task boundary — it does not extend exactly-once to external sinks; DB writes still need idempotency keys. |
| Session vs Tumbling windows | Session windows are the correct abstraction for user activity sessions with variable inactivity gaps; tumbling windows are for fixed-interval metrics (e.g., orders per minute). |
What's Next
Part 5: Saga Choreography and Event-Driven Workflow Patterns addresses the distributed transaction problem that appears when a business operation spans multiple services — each with its own database, its own failure modes, and no shared transaction coordinator. Saga choreography, compensating transactions, and the process manager pattern keep multi-service workflows consistent without two-phase commit.
This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.