Webhook Delivery Architecture: Retries, Signing, Idempotency

Q: Why should webhooks never be sent synchronously inside the request handler?

Because it couples your system's health to every consumer's health and turns transient failures into permanent data loss. If you POST to the consumer inside the handler that produced the event, a slow consumer holds your request open and ties up your resources, and a consumer that is down or rate-limiting makes your own endpoint slow or error — one flaky integrator can degrade your whole API. Worse, if the POST fails and you have no durable record of the event, it is simply gone: you cannot retry what you never stored. The correct shape is to capture the event synchronously into durable storage (ideally a transactional outbox committed in the same transaction as the state change) and deliver it out-of-band with a separate worker fleet. That decouples your latency and availability from consumers entirely, guarantees the event survives consumer outages, and lets you retry on your own schedule. Synchronous delivery is the single most common webhook mistake and the root cause of both stalls and lost events.

Q: What retry policy should a webhook system use?

Exponential backoff with jitter, a bounded number of attempts, and a dead-letter queue at the end. A typical schedule retries after roughly 10 seconds, 1 minute, 5 minutes, 30 minutes, 2 hours, and 5 hours, spread over about 24 hours and often a dozen attempts total, so a consumer that is briefly down recovers automatically without you giving up too early. Jitter — randomising each delay slightly — is essential so that when a consumer comes back online it is not hit simultaneously by every producer's backlog in a thundering herd. Each individual delivery uses a short connect and read timeout, on the order of 5 to 10 seconds, so a consumer that takes 30 seconds is treated as a failure rather than allowed to tie up a delivery worker. When the schedule is exhausted, the event moves to a dead-letter queue that raises an alert and exposes a manual or automatic replay control, so a failed delivery is visible and recoverable rather than silently lost. Pair this with a per-endpoint concurrency cap or circuit breaker so one broken consumer cannot starve delivery for everyone else.

Q: How do I secure webhooks against forgery and replay attacks?

A webhook endpoint is a public URL that receives state-changing events, so you must authenticate the producer or anyone can forge callbacks like "payment succeeded". The standard, used by Stripe, GitHub, and the Standard Webhooks specification, is an HMAC-SHA256 signature over the raw request body plus a timestamp, sent in a header, using a per-endpoint shared secret. Four rules make it sound. First, sign the raw bytes, not parsed-then-reserialized JSON, because re-serialization changes the bytes and breaks verification. Second, include a timestamp and have the consumer reject messages outside a small window (around five minutes), so a captured request cannot be replayed later. Third, use a constant-time comparison when checking the signature to avoid timing side channels. Fourth, store the signing secret properly: per endpoint, rotatable, held in a secrets manager, and never embedded in code or the URL. Treat the webhook sender as a workload identity with a scoped, rotatable credential. Together these stop both forgery (no valid signature without the secret) and replay (old captured requests fall outside the timestamp window).

Q: Why must webhook consumers be idempotent, and whose job is it?

Because reliable delivery is at-least-once: over an unreliable network you cannot guarantee exactly-once delivery, so to avoid ever losing an event you accept that some events will be delivered more than once — for example when a consumer processes a request but its acknowledgement is lost, triggering a retry. If the consumer is not idempotent, that duplicate becomes a double charge, a duplicate shipment, or a doubled counter. The responsibility is shared. The producer's job is to make idempotency possible: assign every event a stable, globally unique id, send the same id on every retry of the same event, and document the idempotent contract. The consumer's job is to use it: record processed event ids and, on seeing one again, acknowledge and skip rather than reprocess. This is what converts at-least-once on the wire into effectively-once processing, which is the best achievable guarantee. Exactly-once is a myth at the transport layer; the practical equivalent is at-least-once delivery plus consumer-side deduplication on a stable id.

Q: Can webhooks guarantee ordering, and how should I handle it?

Global ordering across a distributed producer is impractical and slow, so the honest answer is to promise as little ordering as you can get away with. Most systems should explicitly tell consumers that events may arrive out of order and include a sequence number or version on every event so consumers can detect and ignore stale updates — this lets you parallelise delivery for throughput and is far more robust than a promise you cannot keep. When some consumers genuinely need order for a given resource, offer per-key ordering at most: serialise delivery for events sharing a key (for example, all events for one account) and do not advance to the next until the prior one is acknowledged, while still delivering different keys in parallel. Promising ordering you cannot actually maintain is worse than promising none, because integrators build assumptions on it that break under retries and failover. Carry versions everywhere regardless, so that even unordered consumers can resolve which update is newest and discard stale ones.

Q: What role does a dead-letter queue play, and what goes in it?

A dead-letter queue (DLQ) is where an event lands after its full retry schedule is exhausted without a successful delivery, and it is what turns "we silently lost an event" into "we have a recoverable record and an alert". Without a DLQ, a consumer that is down for longer than your retry window simply loses every event aimed at it, with no trace. With one, those events are preserved with their full context — payload, target endpoint, attempt history, last response — and the DLQ raises an alert so an operator knows delivery is failing, and exposes a replay control (manual or automatic) to redeliver once the consumer is healthy. The DLQ also feeds delivery visibility: combined with a per-attempt delivery log, it lets support answer the question integrators always ask, "did event X reach customer Y, and if not, why?". Treat the DLQ as a first-class part of the pipeline with monitoring and a replay UI, not an afterthought, because the day you need it is the day a major consumer has an outage and you must prove no data was lost.

Q: Why is signing the raw body (not re-serialized JSON) so important?

Because HMAC signatures are computed over exact bytes, and JSON has many byte-level representations of the same logical object. If the producer signs the raw bytes it sends but the consumer parses the JSON into an object and then re-serializes it before verifying, the re-serialized bytes will almost always differ — key ordering changes, whitespace changes, number and unicode escaping changes — and the recomputed signature will not match, so verification fails for legitimate messages. The same breakage happens in reverse if the producer signs a re-serialized form. The rule is therefore that both sides operate on the raw request body exactly as transmitted: the producer signs the bytes it puts on the wire, and the consumer must capture the raw body (before any JSON middleware parses it) and verify the signature against those exact bytes before parsing. Many webhook integration bugs trace to a framework that auto-parses the body and discards the raw bytes; the fix is to preserve and verify the raw payload first, then parse. This is a small implementation detail with an outsized failure rate, which is why every mature webhook spec calls it out explicitly.

Q: What is a sensible maturity path for a webhook system?

Stage 0 is a synchronous POST in the request handler with no retries, where events are lost on any failure and slow consumers stall your API. Stage 1 moves delivery off a queue with basic retries, which is better but still loses events between the business commit and the enqueue if there is no durable capture, and usually has no signing. Stage 2 is the level most teams should reach: a transactional outbox so events are captured atomically with the state change, exponential backoff with jitter plus a dead-letter queue so failures are recoverable, and HMAC signing with a replay window so payloads are tamper- and replay-proof. Stage 3 adds per-endpoint concurrency caps and circuit breakers so one broken consumer cannot starve the rest, full delivery logs with a replay UI, stable event ids with a documented idempotency contract, and secret rotation. Stage 4 adds per-key ordering where promised with versioning everywhere, partner-facing delivery dashboards, automatic DLQ replay, and signed, audited delivery as a compliance artifact for regimes like DORA, NCSC, and MAS guidance. Most teams ship Stage 0 or 1 and discover the gaps through a lost-event incident or a forged callback; the systems integrators trust are at Stage 2 and above from the start.