Webhooks

A webhook is an untrusted HTTP request that claims to be an event. Almost every webhook bug is one of four things: the signature was checked against a re-serialized body, the handler did its work before acknowledging, the same event arrived twice, or events arrived out of order.

When this fires

Building or reviewing a webhook receiver, integrating a provider's events, or designing outbound event delivery to consumers. It does not fire for an internal queue consumer — that has no HTTP trust boundary, though its idempotency requirements are the same.

Procedure — receiving

1. Verify before you parse. Compute the signature over the raw request body exactly as received, before any JSON decode or middleware re-serialization, and compare in constant time. Check the signed timestamp against a tolerance window so an old capture cannot be replayed. A request that fails verification gets a 4xx and nothing else — no logging of the payload as if it were real, no side effects. Never identify the sender by a field inside the payload.
2. Confirm the secret's source. Per-endpoint secret from configuration, never committed, and the verification path must fail closed when the secret is missing rather than skipping the check.
3. Acknowledge fast, work later. Persist the raw body and headers, return 2xx, and process asynchronously. A handler that does its work inline will eventually exceed the sender's timeout, the sender will retry, and you will get duplicates on top of the original that actually succeeded.
4. Deduplicate on the provider's event id. A persisted record with a unique constraint is the arbiter; a read-then-write check is a race, not a guard. At-least-once delivery is the norm — a second delivery of an event you already applied must be a no-op that still returns 2xx.
5. Do not trust arrival order. Order by the event's own sequence number, version, or emitted timestamp, and ignore an event whose version is older than the state you already hold. Where the payload is thin or ordering matters more than latency, treat the event as a signal and fetch current state from the provider's API instead of applying the payload.
6. Make the effect idempotent, not just the dispatch — see idempotency-and-retries. Dedupe keeps you from processing twice; idempotent effects keep you correct when it happens anyway.
7. Use status codes to mean what the sender thinks they mean. 2xx: delivered, stop retrying. 4xx: permanent — the sender will not retry and may disable the endpoint, so return it only for events you will never be able to process. 5xx: retry me. Never return 2xx to silence a failure you have not recorded, and never return 5xx for a malformed event that will fail identically forever.
8. Dead-letter after the retry budget, keeping the raw body, headers, receipt time and failure reason. Provide a deliberate replay path. Replaying events into a production system is an outward-facing action with real side effects: stop and ask before running one.
9. Log without secrets. Event id, type, verification result, outcome. Not the signature header, not tokens or card data inside the payload.

Procedure — sending

10. Publish the contract before the first delivery. Event types and their payload schemas, a stable event id, an emitted timestamp, the signature scheme and which bytes it covers, the delivery headers, the retry schedule, and — stated plainly — the ordering guarantee you actually offer, which is usually none. Promise at-least-once and tell consumers to be idempotent.
11. Sign every delivery with a per-endpoint secret, include the timestamp in the signed material, and support two active secrets so a consumer can rotate without dropping events.
12. Retry with exponential backoff plus jitter, a capped attempt count, and a per-endpoint concurrency limit so one slow consumer cannot drain your workers. Disabling a customer's endpoint after sustained failure is a customer-visible action — notify, and ask before doing it by hand.
13. Treat the destination URL as hostile. It is user-supplied: block private and link-local address ranges, resolve and check the address actually connected to, refuse redirects to internal hosts, set connect and read timeouts, and cap the response size you read. This is the SSRF surface of your whole cluster.
14. Version payloads additively. Adding a field is safe only if consumers were told to ignore unknown ones; removing or re-typing a field needs a new event type or version.

Checklist

• Signature verified on raw bytes, constant time, with a timestamp window
• Missing or misconfigured secret fails closed
• Endpoint acknowledges before doing the work, within the sender's timeout
• Duplicate delivery of the same event id is a no-op, enforced by a unique constraint
• Out-of-order and stale events are detected by version or timestamp, not ignored
• 2xx / 4xx / 5xx each returned for the case the sender interprets them as
• Dead-letter store keeps raw body and headers, with a replay path that asks first
• Outbound: contract published, deliveries signed, backoff bounded, destination URL restricted
• Verified against a real signed delivery, or stated as only exercised with a sample payload

Failure handling

• Signature fails against a known-good delivery — suspect body mutation first: a framework that parses and re-serializes, a proxy that rewrites encoding, trailing-newline handling. Do not "solve" it by weakening or skipping verification; an endpoint that accepts unsigned requests is a remote write primitive for anyone who learns the URL.
• Events arrive but nothing happens — check the async path, not the endpoint. A 2xx from a receiver whose queue consumer is dead looks perfect from outside.
• Duplicate side effects in production — this is the dedupe or idempotency gap, not the provider misbehaving. At-least-once means exactly what it says.
• Cannot receive a real delivery locally — say the handler was exercised with a constructed payload and that live delivery is unverified. A tunnelled or sandbox delivery is better evidence; a unit test with a hand-made signature is the weakest and must be labelled as such.
• Provider's documented headers or schema are uncertain — read their current documentation rather than assuming. Header names, signature formats and tolerance windows differ per provider and change.

Evidence to report

The verification path and what bytes it covers. One delivery you actually observed — event id, type, status returned, elapsed time to acknowledge. The duplicate test: the same event id delivered twice, and proof of one effect. An out-of-order or stale event, and what the handler did with it. For outbound work, the published contract and the retry schedule's actual numbers. And which of these were executed against a real signed delivery versus only constructed in a test — never let "the webhook works" stand in for either.