Siddhant Deval
Siddhant Deval
backend22 min read

DynamoDB at Scale: Partition Internals, GSI Backpressure & ACID Transactions

DynamoDB's performance ceiling is determined at schema design time — a partition key with insufficient cardinality creates a hot partition that table-level throughput scaling cannot fix, because the limit is enforced per-partition at the storage node. This article covers partition key hashing, per-partition hard limits, hot partition detection and salt-key mitigation, GSI backpressure propagation, ACID transactions, and Global Tables conflict resolution.

DynamoDB at Scale: Partition Internals, GSI Backpressure & ACID Transactions

Every AWS primitive is a tradeoff surface, not a feature toggle. DynamoDB's "scale to any workload" promise has a hard asterisk: per-partition hard limits. A table provisioned at 10,000 WCU cannot write faster than 1,000 WCU to a single partition key, because that limit is enforced at the storage node — not at the table control plane. This is why provisioning more capacity does not fix hot-partition throttling. This article covers the physical storage model that produces this constraint, the GSI backpressure failure mode that causes base table write throttling from an under-provisioned index, and the ACID transaction semantics that enable financial-grade consistency in a non-relational system.

Prerequisite chain: Document & NoSQL Databases: MongoDB & DynamoDBPart 7: Access-Pattern-First Modeling → this article.


1. Partition Internals — The Storage Model

DynamoDB stores data in partitions — independently managed storage and compute units. Understanding the partition model is the prerequisite for understanding every performance and cost constraint in DynamoDB.

1.1 Partition Key Hashing

DynamoDB uses an internal hashing algorithm to map partition key values to physical partitions. The hash distributes keys across partitions to balance load. The key insight: all items with the same partition key value land on the same partition, served by the same storage node.

Hash function: partition_key_value → partition_node

"USER#u001" → partition_node_A
"USER#u002" → partition_node_B
"USER#u003" → partition_node_C
"STATUS#PENDING" → partition_node_A  ← ALL pending orders hit the SAME node

If 100,000 orders all share STATUS#PENDING as their partition key, all 100,000 writes route to partition_node_A. Even if the table has 10,000 WCU provisioned, partition_node_A is hard-capped at 1,000 WCU.

1.2 Per-Partition Hard Limits

Limit Value Nature
Storage per partition 10 GB Hard limit — DynamoDB automatically splits at 10GB
Read Capacity Units per partition 3,000 RCU Per storage node — cannot be increased
Write Capacity Units per partition 1,000 WCU Per storage node — cannot be increased
Maximum item size 400 KB Per item — binary blobs belong in S3
Crucial Requirement

The per-partition WCU and RCU limits are hard constraints enforced at the storage node level. Increasing table-level provisioned capacity or switching to On-Demand mode does NOT increase these limits. The only mitigation for a hot partition is to distribute writes across more partition key values.

1.3 Hot Partition Detection

Before mitigating, detect. A hot partition signature in CloudWatch:

TYPESCRIPT
// CloudWatch Insights query: detect hot partition by divergence between consumed and provisioned
fields @timestamp, @message
| filter @message like "ProvisionedThroughputExceededException"
| stats count(*) as throttleCount by bin(5m)
| sort throttleCount desc

// Also monitor these metrics per-table:
// 1. SuccessfulRequestLatency P99 — spikes when partition nodes are stressed
// 2. ThrottledRequests — non-zero = throttling occurring
// 3. ConsumedWriteCapacityUnits vs ProvisionedWriteCapacityUnits divergence
//    → consumed << provisioned but still throttling = hot partition

The diagnostic that confirms hot partition (vs general over-provisioning): ConsumedWriteCapacityUnits is low (e.g., 200 WCU average) but ThrottledRequests is high. The table has capacity — the problem is concentration on one partition.

1.4 Write Sharding — Synthetic Salt Keys

The mitigation for hot write partitions: add synthetic cardinality to the partition key with a salt suffix, then scatter-gather reads across shards.

TYPESCRIPT
const SHARD_COUNT = 10 // Number of write shards

// ❌ Hot partition: all PENDING orders → one partition
const hotItem = { PK: 'STATUS#PENDING', SK: orderId, ...orderData }

// ✅ Salted partition key: 10 shards spread writes across 10 partitions
function getShardedKey(orderId: string): string {
  // Deterministic shard assignment: same orderId always gets same shard
  const hash = parseInt(createHash('md5').update(orderId).digest('hex').slice(0, 8), 16)
  const shard = hash % SHARD_COUNT
  return `STATUS#PENDING#${shard}`
}

const shardedItem = {
  PK: getShardedKey(orderId), // e.g., "STATUS#PENDING#3"
  SK: orderId,
  ...orderData
}

// Reading all pending orders: scatter-gather across all shards
async function getAllPendingOrders() {
  const shardQueries = Array.from({ length: SHARD_COUNT }, (_, i) =>
    docClient.send(new QueryCommand({
      TableName: 'AppTable',
      KeyConditionExpression: 'PK = :pk',
      ExpressionAttributeValues: { ':pk': `STATUS#PENDING#${i}` }
    }))
  )
  const results = await Promise.all(shardQueries)
  return results.flatMap(r => r.Items ?? [])
}
Performance / Safety Warning

Write sharding trades write scalability for read complexity. A scatter-gather read across 10 shards issues 10 parallel queries and merges results in the application. This is acceptable for infrequent bulk reads (fulfillment queue drain, admin dashboard) but adds latency and cost for high-frequency individual reads. Model both the write improvement and the read cost before applying sharding.


2. GSI Backpressure — The Upstream Write Throttle

This is the most dangerous DynamoDB failure mode because it is invisible: a GSI under-provisioned relative to the base table write rate throttles base table writes, not GSI reads.

2.1 The Failure Chain

Scenario: AppTable writes 1,000 WPS. GSI1 provisioned at 10 WCU.

Every base table write that updates an item with GSI1 key attributes
  → triggers a GSI1 write
  → GSI1 is provisioned at 10 WCU → can only handle 10 writes/second
  → at 1,000 WPS, GSI1 is overwhelmed

Effect: DynamoDB throttles the BASE TABLE write
  → ProvisionedThroughputExceededException on PutItem/UpdateItem calls
  → Error surfaces in the application as a base table write failure
  → CloudWatch shows base table ThrottledWrites = high
  → Engineers increase BASE TABLE WCU → no improvement (GSI is the bottleneck)
  → Root cause: GSI1 capacity, not base table capacity

2.2 Monitoring and Mitigation

TYPESCRIPT
// Required CloudWatch alarms — one per GSI, not just per table
new Alarm(this, 'GSI1WriteThrottle', {
  metric: new Metric({
    namespace: 'AWS/DynamoDB',
    metricName: 'WriteThrottleEvents',
    dimensionsMap: {
      TableName: 'AppTable',
      GlobalSecondaryIndexName: 'GSI1', // ← Must specify the GSI name
    },
    statistic: 'Sum',
    period: Duration.minutes(1),
  }),
  threshold: 0,
  evaluationPeriods: 1,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  alarmDescription: 'GSI1 write throttling is causing base table write failures',
})

// GSI capacity must be provisioned at the same rate as the base table
// For On-Demand mode: GSI capacity scales with the table automatically
// For Provisioned mode: set GSI RCU/WCU to at least the expected write rate per item
Crucial Requirement

On-Demand mode eliminates GSI backpressure — DynamoDB scales GSI capacity automatically with table write throughput. For Provisioned mode, the GSI WCU must be sized to match the write rate of items that carry the GSI key. A GSI on a low-cardinality attribute (status field with 3 values) concentrates writes and is a hot-partition risk even in On-Demand mode.


3. ACID Transactions

DynamoDB supports ACID transactions via TransactWriteItems (up to 25 write operations) and TransactGetItems (up to 25 read operations) across any number of tables in the same account and region.

3.1 TransactWriteItems — Cross-Item Consistency

TYPESCRIPT
// Use case: order placement — must atomically:
// 1. Create the order item
// 2. Decrement product inventory
// 3. Create the ledger entry
// All three succeed, or none of them succeed

await docClient.send(new TransactWriteCommand({
  TransactItems: [
    {
      Put: {
        TableName: 'AppTable',
        Item: {
          PK: `ORDER#${orderId}`,
          SK: 'METADATA',
          status: 'PENDING',
          total: 149.99,
          userId,
        },
        ConditionExpression: 'attribute_not_exists(PK)', // Idempotency guard
      }
    },
    {
      Update: {
        TableName: 'AppTable',
        Key: { PK: `PRODUCT#${productId}`, SK: 'INVENTORY' },
        UpdateExpression: 'SET inventory = inventory - :qty',
        ConditionExpression: 'inventory >= :qty', // Prevent negative inventory
        ExpressionAttributeValues: { ':qty': quantity }
      }
    },
    {
      Put: {
        TableName: 'LedgerTable', // Cross-table transaction
        Item: {
          PK: `LEDGER#${orderId}`,
          SK: `DEBIT#${Date.now()}`,
          amount: -149.99,
          reason: 'order_payment',
        }
      }
    }
  ],
  // Idempotency token: safe to retry with same token — duplicate has no effect
  ClientRequestToken: idempotencyKey,
}))
Crucial Requirement

TransactWriteItems consumes 2× WCU per item — once for the write operation and once for the transactional overhead. A 3-item transaction costs 6 WCU regardless of item size. At high write rates, this doubles the effective write cost. Use transactions only when cross-item consistency is a genuine business requirement — not as a convenience wrapper for multi-item writes that do not require atomicity.

3.2 Conditional Writes — Idempotent Alternatives to Atomic Counters

TYPESCRIPT
// ❌ Atomic counter — NOT idempotent
// On a retry, the counter increments TWICE for one logical operation
await docClient.send(new UpdateCommand({
  TableName: 'AppTable',
  Key: { PK: 'PRODUCT#p1', SK: 'INVENTORY' },
  UpdateExpression: 'ADD inventory :delta',
  ExpressionAttributeValues: { ':delta': -1 }
  // Problem: on network timeout, the caller retries → ADD -1 fires TWICE
  // Inventory decremented by 2 for one purchase
}))

// ✅ Conditional write — idempotent optimistic concurrency
// If inventory has already changed (another write snuck in), the condition fails safely
await docClient.send(new UpdateCommand({
  TableName: 'AppTable',
  Key: { PK: 'PRODUCT#p1', SK: 'INVENTORY' },
  UpdateExpression: 'SET inventory = :newValue',
  ConditionExpression: 'inventory = :expectedValue',
  ExpressionAttributeValues: {
    ':expectedValue': currentInventory,  // The value we read
    ':newValue': currentInventory - 1,   // What we want it to be
  }
  // On retry: if the write already succeeded, inventory = currentInventory - 1 ≠ currentInventory
  // → ConditionalCheckFailedException → treat as success (the intended write happened)
  // On concurrent write: another thread changed inventory → ConditionalCheckFailedException
  // → treat as conflict → re-read and retry with fresh expectedValue
}))

4. DynamoDB Streams and Lambda Integration

TYPESCRIPT
// CDK: DynamoDB Streams with Lambda ESM and filter expressions
import { Table, StreamViewType } from 'aws-cdk-lib/aws-dynamodb'
import { DynamoEventSource, StartingPosition } from 'aws-cdk-lib/aws-lambda-event-sources'
import { FilterCriteria, FilterRule } from 'aws-cdk-lib/aws-lambda'

const table = new Table(this, 'AppTable', {
  stream: StreamViewType.NEW_AND_OLD_IMAGES, // Best for audit; most expensive (full payloads)
})

// ESM filter: only invoke Lambda for INSERT events on ORDER items
// Eliminates unnecessary Lambda invocations for other entity types
processorFn.addEventSource(new DynamoEventSource(table, {
  startingPosition: StartingPosition.TRIM_HORIZON,
  batchSize: 100,
  bisectBatchOnFunctionError: true,
  maxRecordAge: Duration.hours(1),
  retryAttempts: 3,
  onFailure: new SqsDestination(dlq),
  filters: [
    FilterCriteria.filter({
      eventName: FilterRule.isEqual('INSERT'),
      dynamodb: {
        NewImage: {
          entityType: { S: FilterRule.isEqual('ORDER') }
        }
      }
    })
  ]
}))
Stream view type Content Use case Cost
KEYS_ONLY PK + SK only Change detection Lowest
NEW_IMAGE New item state Event sourcing Medium
OLD_IMAGE Previous state Soft deletes, undo log Medium
NEW_AND_OLD_IMAGES Both states Full audit trail, diff detection Highest

5. On-Demand vs Provisioned — Cost Crossover

Attribute On-Demand Provisioned + Auto Scaling
Billing model Per request ($1.25/million WRU, $0.25/million RRU) Per hour ($0.00065/WCU-hr, $0.00013/RCU-hr)
Zero-traffic cost $0 Minimum provisioned hours × rate
Traffic spike handling Automatic Delayed by Auto Scaling (3–15 min lag)
Cost crossover Wins for spiky/unpredictable traffic Wins for steady, predictable traffic
Typical break-even ~200 WCU equivalent sustained ~200 WCU sustained 24/7
Partition mechanics: Table → hash ring → physical partitions (per-partition limits labeled) → hot partition in red distributing to green partitions via salt sharding
Partition mechanics: Table → hash ring → physical partitions (per-partition limits labeled) → hot partition in red distributing to green partitions via salt…
Comparison matrix: GSI vs LSI vs Sparse Index vs Table Scan across 7 criteria: consistency, creation constraints, cost, write amplification, cardinality requirements, backpressure risk, projection flexibility
Comparison matrix: GSI vs LSI vs Sparse Index vs Table Scan across 7 criteria: consistency, creation constraints, cost, write amplification, cardinality requ…

Summary

Concept Rule
Per-partition limits 1,000 WCU / 3,000 RCU per partition — table capacity cannot override these
Hot partition detection Consumed WCU << Provisioned WCU + ThrottledRequests > 0 = hot partition, not under-provisioning
Write sharding Synthetic salt key distributes writes; scatter-gather adds read complexity
GSI backpressure Under-provisioned GSI throttles base table writes — alarm on GSI metrics, not only table metrics
TransactWriteItems 2× WCU per item, 25 item max — use for genuine financial-grade consistency only
Conditional writes Idempotent optimistic concurrency — correct alternative to atomic counters on retry paths

What's Next

In Part 9: Messaging Foundations — SQS, SNS & Fan-Out Patterns, we shift from the data layer to the messaging layer: why SNS and SQS solve completely different problems, what happens when you use SNS as a work queue, and how the visibility timeout is the mechanism behind SQS's at-least-once delivery guarantee.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#DynamoDB#Partition Internals#Hot Partition#GSI#ACID#Global Tables#AWS
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.