Idempotency in Distributed Systems: Safe Retries and the Idempotency Key Pattern (2026)

Q: What does it mean for an operation to be idempotent?

An idempotent operation produces the same result and side effects whether it is executed once or multiple times with the same inputs. In the mathematical sense, f(f(x)) = f(x). In distributed systems, a payment charge operation is idempotent if charging a customer twice with the same idempotency key results in only one charge being applied. A database write operation is idempotent if inserting the same row twice results in only one row in the table. HTTP GET, PUT, and DELETE are defined as idempotent by the HTTP specification; POST is not. Idempotency is a correctness requirement in any system where network failures, message re-delivery, or client retries can cause the same operation to be triggered more than once.

Q: What is an idempotency key and how should clients generate it?

An idempotency key is a unique identifier that represents a specific logical operation, generated by the client before the first attempt. The server uses this key to recognise that a request is a retry of an operation it has already processed and returns the cached result without re-executing the operation. Clients should generate idempotency keys as UUIDs or ULIDs using a cryptographically secure random source — never use predictable values like incrementing counters or timestamps. The key must be generated before the first request (not during), persisted in the client's state (not in memory only), and reused for every retry of the same logical operation. A new key should be generated for each new logical operation, even if it involves the same parameters as a prior operation.

Q: Why is Redis used for idempotency key storage rather than the application database?

Redis is preferred for idempotency key storage because of three properties: it supports TTL natively (keys automatically expire after the retry window closes, avoiding unbounded table growth), it provides sub-millisecond read performance (the idempotency key lookup adds minimal latency to each request), and it supports atomic SET NX (set if not exists) which prevents race conditions when two concurrent requests arrive with the same key. The application database can also be used — and should be as the second layer of defence via unique constraints — but using Redis as the first-line check keeps the idempotency lookup off the critical path of the main database transaction. If Redis is unavailable, the fallback is the database unique constraint, which provides correctness at the cost of a failed insert on duplicates.

Q: How do message queues relate to idempotency?

Most production message queues — Kafka, RabbitMQ, Amazon SQS Standard — use at-least-once delivery semantics. This means a message that was processed successfully but not acknowledged (because the consumer crashed) will be redelivered. The consumer must handle this redelivery without applying the same operation twice. The standard approach is to store the message ID in the database before processing, inside the same transaction as the actual business logic. If the message has already been processed (its ID is in the table), the consumer skips processing and acknowledges the message. If the processing fails, the transaction rolls back and the message is reprocessed on the next delivery. Kafka's transactional API provides exactly-once semantics within a Kafka cluster, but at higher operational complexity — consumer-side idempotency with database unique constraints is the more widely applicable approach.

Q: What is the difference between idempotency and exactly-once processing?

Exactly-once processing means an operation is guaranteed to execute precisely once, regardless of retries or failures. This is technically impossible to guarantee in a distributed system without using distributed transactions — which are expensive and often impractical. Idempotency is a weaker but achievable guarantee: the operation may execute multiple times, but only the first execution has observable side effects. A payment processed with idempotency will charge the customer's card on the first execution and return the cached result on all subsequent retries — the customer is charged once. Exactly-once processing would prevent the charge from even being attempted a second time. In practice, idempotency achieves the same business outcome as exactly-once (one charge, one inventory deduction, one order created) at far lower operational cost.

Q: How do you handle idempotency in a multi-step distributed transaction (saga)?

Each step in a saga must be independently idempotent because the saga orchestrator will retry failed steps. The key is ensuring each step has a stable idempotency key that is derived from the saga's context — not generated fresh on each retry attempt. For example, the inventory reservation step for order ord-123 uses the key reservation:ord-123:prod-456, and the payment step uses the key payment:ord-123:attempt-1. When the orchestrator retries a timed-out step, it uses the same idempotency key, and the step returns the cached result if it already completed. Without stable idempotency keys at each step, a saga under retry conditions will create duplicate reservations, charges, or confirmations. The saga orchestrator itself must also be idempotent — processing a saga completion or failure event multiple times should produce the same final state.

Q: Should idempotency keys be scoped to a user or global?

Idempotency keys should be globally unique — or namespaced by API key / user ID to prevent cross-user collisions. If two different users generate the same UUID (astronomically unlikely but theoretically possible), a global key store would incorrectly return one user's result to the other's retry. The safer implementation namespaces the key: the Redis key is stored as idempotency:{apiKey}:{idempotencyKey} or idempotency:{userId}:{idempotencyKey}. This also provides a security boundary: a user cannot replay another user's operation by guessing their idempotency key. The idempotency key should be treated as a credential — it should be opaque (UUID), short-lived (24-hour TTL), and scoped to a single authenticated session or API client.

Q: What should happen when an in-progress request is retried before it completes?

When a request is received and marked IN_PROGRESS in the idempotency key store, a concurrent retry with the same key arrives before the first request has completed. The correct response is HTTP 409 Conflict with a message indicating that a request with this idempotency key is already being processed — the client should wait and retry after a backoff period. The alternative — queuing the retry behind the in-progress request and returning its result — is more user-friendly but complex to implement safely. The simpler 409 response is the standard for most payment and e-commerce APIs (including Stripe). Clients implementing retry logic should handle 409 with a longer backoff (5–10 seconds) rather than immediate re-retry, since the in-progress operation needs time to complete. Once the first request completes and stores its result, subsequent retries will find the completed result and return immediately.

Satyam Kumar

返回博客

ai-architecture

Idempotency in Distributed Systems: Safe Retries, Deduplication, and the Idempotency Key Pattern (2026)

By Satyam KumarApril 17, 202615 min read

idempotency distributed systems microservices patterns safe retries idempotency key deduplication at-least-once delivery message queue payment idempotency api design redis software architecture

Idempotency in Distributed Systems: Safe Retries, Deduplication, and the Idempotency Key Pattern (2026)

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

Idempotency in Distributed Systems: Safe Retries, Deduplication, and the Idempotency Key Pattern (2026)

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment