System Design Fundamentals

Idempotency

Phone retries; the ledger already charged. The spinner never cleared after Pay, the network flaked, and the app sent the same charge again. Without care, the card is billed twice: once for the success the user never saw, and once for the retry that looked like a new request.

means that retry does not create a second . Payment platforms live on ; queues survive because handlers tolerate delivery and aim for through dedupe, not once-on-the-wire delivery.

Retry journey

Reuse the same key on every retry of this logical charge.

Server records processing

Insert the key as processing before any lasting side effect runs.

Create the charge or order exactly once under that key.

Replay on timeout

If the response is lost, look up the key and return the stored result.

What actually happens

The dangerous case is not a clean duplicate click. It is success on the server followed by a lost response. The client retries with the same . The server looks up the key and replays the stored response instead of charging again. That is the whole idea: treat retries of one logical operation as one operation.

Concurrent duplicates are the other hard case: two in-flight requests share a key. Only one may run the ; the loser waits, gets 409 while status is processing, or eventually receives the same replayed result. Key scope matters: keys are usually unique per tenant and operation type, not globally across unrelated APIs.

GET, PUT, and DELETE are naturally closer to idempotent. POST creates resources, so it needs a key, a unique constraint, or a . Messaging and workers usually assume delivery and aim for through dedupe, not once-on-the-wire delivery.

The atomic-boundary trap: if you charge the card then crash before storing the key result, a retry can double-charge. The dedupe record and the business write must share one transaction, one conditional write, or a durable reconciler. After expires, a late retry is treated as a new operation, so size TTL to the client retry budget (see Numbers to know below).

The sequence below is the lost-response path from the hero: first attempt completes, the packet never arrives, and the retry replays instead of charging again.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    participant Client
    participant API
    participant Store as Idempotency store
    participant DB as Business DB
    Client->>API: POST /charges key=abc
    API->>Store: insert key abc as processing
    API->>DB: create charge in same transaction boundary
    API->>Store: save response as completed
    API--xClient: response lost
    Client->>API: retry POST /charges key=abc
    API->>Store: lookup key abc
    Store-->>API: completed response
    API-->>Client: replay original response

How to choose an idempotency design

Once you know retries are inevitable, three questions decide whether you need a key store, a domain unique constraint, or both.

1. Is the operation a mutation with side effects? Reads usually need caching, not keys. Payments, orders, emails, and webhooks need stable operation identity so a timeout cannot create a second lasting change.

2. Does the domain already have a natural unique key? Prefer a when it exists (email for signup, order_id + attempt for a payment, file hash for uploads). Otherwise generate a synthetic UUID once per logical attempt and reuse it on every retry.

3. How long can clients retry? Set to cover that window: longer for money (often 30–90 days), shorter for chatty APIs (often 24–72 hours). A TTL shorter than the retry budget turns a late retry into a new operation.

Simple rule of thumb

Store key + + status + response. Return 409 while processing. Replay when completed. Reject same key with a different hash. Never mint a new key on every retry of the same logical operation. In simple terms: delivery plus idempotent processing (with response replay) is how production systems get , not once-on-the-wire delivery.

From problem to mechanism

Here is the Pay spinner again, slowed down so you can see where a naive POST fails and what the store changes.

Problem. Clients retry after timeouts; queues redeliver; users double-click. Each retry looks like a fresh request unless the server recognises one logical operation.

Naive system. Treat every POST as new work. Disable the Pay button in the UI and hope the network never drops a success response. There is no , no stored response to replay, and no shared transaction between the charge and the dedupe record.

Break it. The card network charges successfully; the response packet is lost; the client retries and charges again. Or two pods both accept the same key while status is still processing and both call the network. Or a late retry arrives after expiry and is treated as a brand-new charge.

Mechanism. Require a scoped key (tenant + operation + key), fingerprint the body with a , mark the row processing before the , then store the final response. Retries wait (409), replay, or get 400 on hash mismatch. Keep the dedupe record and business write in one transaction boundary (or a durable reconciler) so crash-after-effect cannot double-charge.

Correctness invariant

For a scoped within , at most one runs; every retry either waits (409/processing), replays the stored response, or rejects a mismatched . The dedupe record and the business write share one transaction boundary (or a durable reconciler). Otherwise crash-after-effect can double-charge. This yields exactly-once effects, not once-on-the-wire delivery.

Numbers to know, and where this shows up

The mechanism above is not optional at payment scale. Here is a concrete checkout load, then the everyday places the same pattern appears (HTTP methods, queues, databases).

Quantitative worked example

Assume 2,000 logical charge attempts per second. If 3% are retried once and a separate 0.5% are retried twice, the request rate becomes 2,000×(1+0.03+2×0.005)=2,080 requests per second. Those 80 extra requests are potential duplicate effects only if the earlier attempts actually succeeded and every retry repeats the charge. State that failure assumption explicitly; network retries alone are not proof that 80 double charges occur.

Idempotency store needs ~2_000 inserts/sec plus lookups; retain 72h (~518M rows) and shard aggressively.
Same key with a different hash must return 400; completed replay should stay under 10ms p99.
While status is processing, return 409 so two pods cannot both call the card network.

HTTP methods

GET, PUT, DELETE are closer to idempotent; POST needs a key or unique constraint.

Payments

charge(order_123, $100) with a key is safe to retry; without one, retry double-charges.

Messaging

Producer id plus sequence lets the broker dedupe; without it, duplicates land in the topic.

APIs

POST /orders with Idempotency-Key: first creates; retry returns the existing order.

Database

INSERT … ON CONFLICT (idempotency_key) DO NOTHING, or return the existing row.

Event processing

Consumer crashes before ack, then redelivery; check already-processed and skip side effects.

Idempotency key storage playbook

Numbers to know above set the insert and retention load; this is the durable shape of the store, how requests flow through it, and which backend fits which workload.

sql

CREATE TABLE idempotency_keys (
  tenant_id        VARCHAR(64) NOT NULL,
  operation_type   VARCHAR(64) NOT NULL,
  key              VARCHAR(255) NOT NULL,
  request_hash     VARCHAR(64) NOT NULL,
  status           VARCHAR(20) NOT NULL, -- processing, completed, failed
  lock_expires_at  TIMESTAMP,
  response_code    INTEGER,
  response_pointer VARCHAR(255),         -- large response stored elsewhere
  response_body    JSONB,                -- small response replay
  created_at       TIMESTAMP NOT NULL,
  expires_at       TIMESTAMP NOT NULL,
  PRIMARY KEY (tenant_id, operation_type, key)
);
CREATE INDEX idx_idempotency_expires ON idempotency_keys (expires_at);

Flow

Receive the request carrying an Idempotency-Key header for this logical attempt.
INSERT INTO idempotency_keys (key, status) VALUES ($1, 'processing') ON CONFLICT (key) DO NOTHING
If the insert succeeded, process the request and persist the final response under that key.
If the insert failed: completed replays stored; processing returns 409; failed may allow retry.

Cleanup and TTL

Retain deduplication records for the documented retry and replay horizon, then delete expired records in indexed batches. Public API products have different retention policies; 24 hours or 90 days are design examples, not universal payment guarantees. Even after an API response cache expires, a durable order or payment-attempt identifier can enforce business uniqueness. Tell clients whether an expired key is rejected or can represent new work.

At scale (1B keys/day)

Storage: 1B × 500 bytes ≈ 500 GB/day before cleanup; with 24h TTL ≈ 500 GB steady state.
Keep an in-memory store for hot keys and let TTL clean expired entries automatically.
Use a relational database when payment keys must survive restarts and audits.
Partition by key hash so inserts and lookups scale horizontally across shards.

Production store design

Store	Best for	Important fields	Risk
Relational table	Payments, orders, reservations	tenant, operation type, request hash, status, response pointer, cleanup cursor	Must partition and clean old rows.
Key-value table	High-volume API dedupe	composite key, TTL, compact response, lock expiry	Eventual reads can allow duplicates unless writes are conditional.
Cache hot path + durable log	Low-latency retries	short TTL cache entry, durable reconciliation event	Cache alone is unsafe after eviction or restart.

Storage estimate

plaintext

100M keys/day × 700 bytes × 3 days retention
≈ 210 GB before replication
With 3 replicas ≈ 630 GB

1B keys/day × 700 bytes × 3 days
≈ 2.1 TB before replication
With 3 replicas ≈ 6.3 TB

Concurrent duplicate flow

plaintext

Request A inserts key as processing with lock_expires_at.
Request B arrives with same key while A is running.
B checks request_hash:
  mismatch -> 400 bad retry
  same hash + processing -> 409 or wait briefly
A completes side effect and stores final response.
B retries and replays stored response.

Atomic-boundary trap

The idempotency record and the must share one transaction, one conditional write, or a durable reconciliation path. Otherwise a crash after the side effect but before storing the result can still create duplicates.

Keys, constraints, and exactly-once effects

With a store shape in hand, the remaining design choices are which key to mint, how to enforce uniqueness in the database, and how queues turn delivery into .

Natural vs synthetic keys

Payment

order_id + payment_attempt_number becomes a stable string like order_123_attempt_1.

User registration

Email as a UNIQUE constraint prevents two accounts for the same address.

Inventory update

order_id + product_id reserves five units once, even if the call retries.

File upload

File content hash deduplicates the same bytes without an extra client header.

Natural keys

Use a when business logic already defines uniqueness. No extra header is needed, so clients stay simpler.

Synthetic (UUID)

The client creates one synthetic identifier when the user begins a logical operation and persists it until the outcome is known. Retries after a timeout, application restart, or transient server error reuse that identifier. A genuinely new purchase receives a new key even if its body looks identical to an earlier purchase. This distinction matters because a hash of request contents alone cannot tell a duplicate delivery from an intentional second order.

Implementing with database constraints

Pattern 1: Idempotency table + business logic in one transaction

sql

BEGIN;
  INSERT INTO idempotency_keys (key, status) VALUES ($1, 'processing');
  -- If duplicate: UNIQUE violation → catch → return stored result
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO payments (...) VALUES (...);
  UPDATE idempotency_keys SET status='completed', response=$2 WHERE key=$1;
COMMIT;

Pattern 2: UPSERT with conflict handling

sql

INSERT INTO orders (idempotency_key, user_id, total, status)
VALUES ('key_abc', 123, 100.00, 'pending')
ON CONFLICT (idempotency_key)
DO UPDATE SET updated_at = NOW()
RETURNING *;
-- First: creates order. Retry: updates timestamp only, returns existing

Pattern 3: Conditional write

javascript

PutItem(
  Item: {pk: "order_123", sk: "payment_1", amount: 100},
  ConditionExpression: "attribute_not_exists(pk)"
)
// First: writes successfully. Retry: ConditionalCheckFailed → order exists

Race condition handling

Enforce a unique key over tenant, operation type, and client operation identifier. Two callers that race to create the same processing record cannot both win the insert; the loser reads the existing state and either waits within a bound, receives an in-progress response, or replays the stored result. A separate “check then insert” is unsafe because both callers can observe absence. Compare a canonical request fingerprint so the same key cannot silently refer to different amounts or recipients.

Exactly-once processing pattern

"Exactly-once" in practice is: delivery + idempotent processing.

Exactly-once log processing

Producer:

yaml

kafka_producer:
  enable.idempotence: true
  # Broker dedup key per partition: producer_id + sequence

Consumer + processing:

Read message → process → commit log offset and database write in one transactional boundary when supported.

Queue + function worker

Queue redelivery is expected when a worker commits work but crashes before acknowledging the message. Insert the message identifier into an inbox and apply the local business mutation in the same database transaction. If the identifier already exists, skip the mutation and acknowledge the duplicate. A nontransactional lookup in processed_messages followed by an external effect is insufficient: two workers can race, or a crash can occur between the effect and the marker.

General pattern

Receive the message or request that may already have been delivered once.
Check the dedup store: already processed returns stored; otherwise process, store, and mark atomically.
Acknowledge the message only after the durable mark exists so redelivery stays safe.

Process + mark-as-processed must be in the same transaction. Otherwise: process → crash → re-process → duplicate side effect.

Payment walkthrough

http

POST /v1/charges HTTP/1.1
Idempotency-Key: charge_order_123_attempt_1
Content-Type: application/json

{"amount":5000,"currency":"usd","source":"tok_visa"}

First request

Create the charge, store key → response, then return 200 with charge id and status succeeded.

Network timeout → retry with same key

Key exists and completed, so return the stored response; no second charge is created.

Different request body with same key

Return 400: idempotency key already used with different params. must match.

javascript

// Best practice for clients
key = generateUUID()  // Before first attempt
for (attempt = 0; attempt < 3; attempt++) {
  response = paymentClient.createCharge({...}, {idempotencyKey: key})
  if (response.ok) break
  // Same key for ALL retries of this logical operation
}

Failure timeline, and what still breaks after the fix

Putting the pieces together, here is the success-then-lost-response path minute by minute, the retry cases that matter in production, and the ways a careful design can still go wrong.

Timeline: success then lost response

T0  Client POST /charges Idempotency-Key=abc; API inserts key as processing
T1  Card network charges $50; API stores charge row + marks key completed
T2  Response packet lost; client times out
T3  Client retries same key=abc
T4  With store: replay stored 200: one charge. Without store / wrong boundary:
    charge succeeds then crash before key write → retry charges again
T5  Multi-layer retries without a shared key mint new keys → double side effects

Counterexample to "POST is fine if the UI disables the button": the dangerous path is success on the server with a lost response.

Retries and timeouts

Timeout scenario (hardest case)

Server processes successfully; response is lost. Client retries with the same key. Server returns the stored response: no duplicate charge.

In-flight retry

Server still processing. Client retries → status=processing → 409 Conflict. Later retry → status=completed → stored response.

Partial failure (double-charge risk)

Charge succeeds at the provider; server crashes before storing the idempotency result. Retry has no stored result → processes again → double charge.

For a local database effect, commit the effect and idempotency result together. An external payment provider is outside that transaction, so a local commit cannot atomically include the remote charge. Persist the payment attempt before calling out, reuse a stable provider-supported idempotency key, and reconcile ambiguous results by that provider reference after a crash. If the provider offers neither deduplication nor a reliable status lookup, the application cannot honestly guarantee one external charge using a local table alone.

Still fails when…

The external charge succeeds, then the process crashes before the key row is marked completed.
Clients mint a fresh key on each retry of the same logical payment attempt.
The same key is reused with a different body, and hash checks are skipped or ignored.

Trade-offs

Longer TTL improves late-retry safety but grows storage and cleanup cost.
Returning 409 for in-flight work is safer than running a second side effect.
Natural keys are simpler when the domain has one; synthetic UUIDs are more general.

Choosing and explaining idempotency

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Every retryable mutation needs a stable operation identity.

Store the request hash and final response so duplicate retries are validated and replayed.

The dedupe record and business side effect must be atomic or reconciled by a durable workflow.

When it belongs

Clients retry POST requests after timeouts.

Queues deliver at least once.

Payments, orders, reservations, emails, or webhooks can cause duplicate side effects.

How it fails

Server crashes after external side effect but before recording the idempotency result.

Client generates a new key for every retry.

Same key is reused with a different request body.

Trade-offs to weigh

Long TTLs improve retry safety but increase storage.

Returning 409 for in-flight work is safer than running duplicate work.

Natural keys are simpler when the domain has one; synthetic keys are more general.

Common misunderstandings

Claiming POST becomes idempotent without a key or unique constraint.

Caching only successful responses and losing failure semantics.

Ignoring cleanup, request hashing, and abuse prevention.

Try it: idempotency key replay

Send a mutating request with a stable idempotency key, then replay the same key after a timeout or client retry. The server should return the original outcome without duplicating side effects.

Idempotency key

Idempotency store

No keys stored yet.

Payment executions: 0
Keys stored: 0
Duplicate charge on replay: No

Submit a payment with an idempotency key. Replays with the same key return the stored response without charging again.

Going deeper

Idempotency

The mechanism and its boundary

Idempotency makes retrying safe by binding repeated attempts to one logical operation. The server should store enough request identity and final response state to return the same result without redoing side effects.

Retry-safe write path

Receive key

Conditional insert

Run effect

Return stored result

Work through the sizing assumptions

Storage estimate: 100M idempotency keys/day * 200 bytes/key record is about 20 GB/day before indexes and replication.

With a 72-hour retention window and 3x replication, 20 GB/day becomes about 180 GB raw replicated storage.

At 10,000 payment attempts/sec with a 1% retry rate, the key store handles 10,000 writes/sec plus about 100 duplicate reads/sec.

When it belongs

Clients or queues retry after timeouts. The operation has non-repeatable side effects such as charging, booking, or sending. The client can provide or derive a stable operation key.

Assumptions that matter

Keys are partitioned by high-cardinality identifiers. The key store uses conditional insert or compare-and-set. Retention matches business retry and dispute windows.

Easy mistakes

Recording only successful operations leaves a second request unable to distinguish an operation still running from one that never started, so persist in-progress and terminal outcomes as needed. Reusing one key with a different request body must produce a conflict, because otherwise the server can return an earlier result for a different intended operation. Removing a key before the documented retry window ends allows a delayed retry to execute again, so retention and client behavior must agree.

How it fails

Response lost after side effect commits, then retry repeats the effect. Concurrent duplicate requests both pass the first existence check. Hot clients generate many keys and overload one partition.

Trade-offs to name

Persisting operation identity makes uncertain retries safer, but adds conditional writes and state to the request path, so the record must be part of the same correctness boundary as the effect it protects. Longer retention covers more delayed retries while increasing storage and cleanup cost, so define the supported retry horizon instead of silently treating an expired key as safe forever. Replaying a stored response gives clients a stable result, but requires preserving the outcome and handling any sensitive response fields under an appropriate retention and access policy.

Wrap-up

You have seen the lost-response failure, the key store mechanism, and the operational signals. Use these exercises to check the ideas stuck, then follow the references for primary sources.

Exercises

Atomic boundary design. Provider charge is outside your DB transaction. Sketch the durable reconciler (or outbox) so a crash after charge but before key completion cannot double-charge on retry.
Key scope conflict. Two APIs share a global key namespace. A refund reuses a charge key by accident. What scoping (tenant + operation_type) and hash check prevent the wrong replay, and what status code do you return?
TTL vs late retry. Clients may retry for 48h; keys TTL at 24h. Quantify the duplicate risk and decide the retention policy for payments vs chat sends.

Topic-specific references

Stripe: Idempotent requests : key reuse, request-hash mismatch, and response replay semantics.
RFC 9110 HTTP methods : which methods are defined as idempotent at the protocol layer.
AWS Builders' Library: Making retries safe with idempotent APIs : concurrent duplicates and atomic boundaries in practice.

Key takeaway

At scale, every mutation should be safe to retry. Scope keys carefully, store key + request hash + response with the business write, size to the retry window, handle concurrent duplicates with 409/wait-then-replay, prefer natural keys when they exist, and treat at-least-once delivery plus idempotent handlers as your path to exactly-once effects.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Scope and fingerprint idempotency keys, coordinate concurrent duplicates, replay responses, choose retention, handle regions, and prevent abuse.

Explain the decision

What if the same idempotency key arrives with a different body?

Read the reasoning

Reject it as a conflict by comparing a canonical request fingerprint; never reinterpret one key as another operation.

Apply it to a production scenario

Two identical requests arrive concurrently before either finishes. What does the store do?

Read one approach

One conditional insert becomes owner; the other waits/polls or receives the same in-progress/result state, never executes independently.

Work through the numbers

At 4,000 operations/s and 24-hour retention, how many dedup records?

Read the worked calculation

345.6 million records; size, partition, expiry, and abuse controls must be planned.

Reason through a failure

What must response replay preserve?

Read the recovery reasoning

A retry receives the same semantic terminal result for that scoped key, including a prior failure that should not re-execute the effect.

Sources and further study

Continue with Back-of-the-Envelope Estimation.