Resources · 54

Webhooks: handle repeats, delays and recovery

Turn a received notification into a unique, traceable business effect recoverable after failure.

· 10 min

Communications racks and network cabling in an empty equipment room AI-generated editorial illustration

What this guide helps achieve

  • Authenticate notifications
  • Separate receipt from processing
  • Prevent duplicate effects
  • Recover blocked events

Quick check

  • Which bytes are signed?
  • When is receipt durable?
  • Can two workers act together?
  • Does the provider guarantee order?
  • How is a lost event found?

Step-by-step method

  1. 01

    Read the provider contract

    Record types, version, account context, signatures, timing and retry rules. Stripe states that delivery ordering is not guaranteed and duplicates can occur; do not apply a Stripe-specific delay to every API.

    Deliverable: dated provider contract.

  2. 02

    Validate before effects

    Check signatures with the library and raw body required by the provider. Limit accepted sizes and types. Test invalid signatures and unexpected account context in an authorised environment without logging secrets.

    Deliverable: denials with no effects.

  3. 03

    Persist before acknowledging

    Separate durable receipt from slow processing. If the queue cannot accept the event, do not report it as retained. Success indicates receipt under your contract, not necessarily completion of the business action.

    Deliverable: received, processing, completed and blocked states.

  4. 04

    Enforce idempotency

    Define event keys and business-effect identity. Test concurrent repeats rather than only sequential delivery. A preliminary check without atomic constraints or locking can allow two workers to create the same effect.

    Deliverable: evidence of effect uniqueness.

  5. 05

    Handle delay and disorder

    Do not assume an old notification describes current state. Consult the source when the contract permits and apply validated business transitions. Retain events that cannot yet be interpreted.

    Deliverable: transitions and reversed-order tests.

  6. 06

    Reconcile and replay

    Prepare an error queue with owner, cause, attempts and closure. Keep duplicate protection during replay. Periodically compare provider and application: successful recovery does not prove no event was missed.

    Deliverable: recovery procedure and discrepancy report.

Four situations for testing the method

Fictional teaching cases. Each situation connects a signal, a check and a decision bounded by evidence; none represents a client outcome.

Fictional situationConcrete checkDecision and limit
A repeated payment event must not duplicate its effect — The same payment notification arrives twice, and processing attempts to activate one subscription twice.Verify authenticity, recognise repeats and reconcile invoice and subscription state with the authorised source.Confirm one business effect and record repetition without copying secrets. An HTTP acknowledgement alone does not establish correct activation.
Separate a send request from delivery — An API accepts an SMS request, and an internal dashboard immediately marks it delivered before delivery status returns.Retain identifiers and distinguish accepted, pending, failed and delivered using received events. Test delayed and repeated notifications.Show the known state with its time and handle uncertainty. An accepted request is not evidence that the person received the message.
An unsubscribe must survive late events — Someone unsubscribes, then an older synchronisation puts their profile back into a campaign segment.Test reversed update order, repeated events and preference checks immediately before authorised sending.Define resolution rules and a trustworthy preference source. Successful synchronisation does not establish correct sending eligibility.
Replay a workflow from its failed step — A workflow creates a record and then fails to notify. A full retry creates a second record.Identify each step’s effects, keys recognising an existing operation and the restart point after partial failure.Resume without repeating a confirmed effect, or use controlled compensation. An automation must make intermediate states explicit.

Reusable worksheet

Complete with your authorised observations. These fields are a working template, not observed results.

FieldInformation to record
EventProvider, account, identifier and version
ReceiptValidated signature and durable timestamp
EffectBusiness key, state and uniqueness evidence
RecoveryCause, attempt, owner and closure

Worked example

Illustrative situation

Fictional example: two workers receive the same confirmation notification during recovery.

Decision and expected evidence

A unique business-effect constraint and transactional state permit one fulfilment while both deliveries remain traceable.

Distinguish the mechanisms

MechanismPurposeCheck or limitation
SignatureValidate origin under the contractDoes not make effects unique
DeduplicationRecognise repeat notificationsAlso check concurrency and business effects
ReconciliationFind omissions and differencesName the source and period

Management indicators

IndicatorWhat it measuresFirst action
Queue ageDelay of unfinished eventsInspect the oldest
Duplicate effectsBusiness actions repeated in errorCorrect atomicity
Source / application gapsAbsent or divergent statesReconcile with closure evidence

Common pitfalls

  • Parsing before signed-body verification
  • Acknowledging before durable receipt
  • Assuming ordered delivery
  • Testing duplicates only sequentially

Frequently asked questions

Does HTTP success mean processing is complete?

It depends on the contract. With asynchronous processing, it should mean durable receipt while business state is tracked separately.

Can exactly-once delivery be assumed?

Design for repeats and failures under the real contract, with unique effects and reconciliation.

Should the entire body be retained?

Only when necessary with suitable access and retention. Minimal traces should support diagnosis and recovery without keeping secrets.

Idempotence and API retry contracts

Separate request retries from notification repeats

An outgoing API request key, a received-event identifier and a business-effect identity serve different roles.

  • Define provider, account, event and effect before selecting a deduplication key.
  • Test concurrent workers and an unknown external-effect outcome.
  • Reconcile business state with the source; a valid signature does not make processing unique.

Compare idempotency contracts →

References reviewed · RFC 9110 · §9.2.2, §10.2.3 · RFC 6585 · §4 · Stripe

Webhooks: signatures and replay

Authenticate a delivery, retain the event and control its effect

A webhook signature answers a specific question: do the received bytes match those authenticated with the expected key? It guarantees neither freshness nor uniqueness nor successful business processing. Separate these decisions in the receipt contract so that the integration can be observed and recovered.

The signature and recovery rules below belong to the cited Stripe and GitHub contracts. The timeline and ten cases are fictional exercises; this page activates no endpoint, real secret or payment mechanism. Use official libraries and test the relevant provider configuration for a real integration.

Fictional empty network gateway with amber fibre links, a black rack and smoked glass partition. AI-generated editorial illustration
Fictional empty network gateway with amber fibre links, a metal rack and smoked glass partition. Generated illustration, unrelated to a real facility or evidence of a security check.

Stripe and GitHub: six differences to retain

Stripe and GitHub: six differences to retain
ControlStripeGitHub
Signature headerStripe-Signature contains t and one or more v1 signatures. The documented active scheme is v1.X-Hub-Signature-256 contains a SHA256 HMAC prefixed with sha256=.
Authenticated messageText timestamp, a period and the raw body: timestamp + "." + body.The raw delivery body. Transport headers are not added to this HMAC message.
Freshness and replayThe timestamp is authenticated. Documentation states a five-minute default tolerance in its libraries; each retry receives a new timestamp and signature.The body HMAC alone provides no freshness proof equivalent to Stripe’s timestamp. Plan duplicate detection and recovery according to the contract.
Identity to trackevent.id identifies repeated delivery. Stripe also describes distinct events that may require correlation using data.object.id and event.type.X-GitHub-Delivery identifies the delivery; requested redelivery retains that identifier. Its value remains outside the body authenticated by the HMAC described here.
Delivery failureLive-mode automatic retries for up to three days with exponential backoff. A manual resend does not automatically cancel these retries.GitHub does not automatically redeliver failures. Arrange manual redelivery or application-managed recovery using the available APIs.
Fast acknowledgementReturn a 2xx quickly, before lengthy work. Durable receipt before acknowledgement is the architectural policy of our exercise.Respond with 2xx within ten seconds according to the documentation. A queue separates receipt from business processing.

Retain three separate identities

Retain three separate identities
IdentityFictional examplePurpose and limit
Transport attemptdelivery_attempt_02One event can arrive several times. An attempt’s timestamp and signature are not a business key.
Provider eventprovider + account + environment + evt_training_001Recognise the same event within its scope. Define retention and recovery behaviour; a bare identifier without context may be insufficient.
Business effecttenant_7 + case_42 + initial_activationPrevent a second initial activation here. The key belongs to this operation; do not discard all future object updates by reducing them to the object identifier alone.
Three bodies, six HMACs: what reformatting changes

Public exercise-only key: training-only-key-not-a-secret. Fixed timestamp: 1792663200, corresponding to 2026-10-22 at 10:00:00 UTC. Strings use UTF-8 with no trailing newline. A and B represent the same JSON object but have different bytes; C changes case_42 to case_43.

Exact bytes of the three bodies
BodyUTF-8 bytesExact string, written as a JSON string
A68"{\"id\":\"evt_training_001\",\"type\":\"evidence.ready\",\"object\":\"case_42\"}"
B73"{\"id\": \"evt_training_001\", \"type\": \"evidence.ready\", \"object\": \"case_42\"}"
C68"{\"id\":\"evt_training_001\",\"type\":\"evidence.ready\",\"object\":\"case_43\"}"
Six digests, two message constructions
Body / schemeMessage usedCalculated hexadecimal HMAC-SHA256
A · GitHubbodyd6ac0a608c901617782f3b7aef64198718e3affd410a219b22b84da008235364
A · Stripe v1timestamp + "." + body0ad9c9f7c0df2d8a8265113e641d92f57ce0dffd703a7cf9f13c4163e21bf6dc
B · GitHubbodyc9c47b1432282b9a3c2cbf4aded2c750deed235ac948dffe248488b84a4af73d
B · Stripe v1timestamp + "." + body0ad8e262a5870766d4cce864f50a5feac65a3004e3c3208184da9cc075c7abd6
C · GitHubbody55162fc08e3cb0efd0eb082978ba61a88cf3624ffc5322f19ab73d91466a616d
C · Stripe v1timestamp + "." + bodyd7f4014489a900399a7049daa3e8f1ce2b208bf95b93c43492cb68f7d6f1b791

These results use the documented message constructions with a publicly displayed key and invented data. They are not authentic Stripe or GitHub deliveries. Different JSON serialization, even if semantically equivalent, must not replace the original body before verification. In production, compare in constant time and manage key rotation according to the selected library and contract.

Stripe tolerance parameter: the value 0 disables recency checking in the libraries described by its documentation. It does not mean “accept only the exact second”. Do not transfer the OAuth/OIDC example’s clock policy to this parameter.

Seven records for one local effect

Fictional scenario on 2026-10-22, all times UTC. Authenticate before admission and persist receipt before responding. An atomic constraint protects initial activation in the same database as its business state. Counts below are calculated for this exercise and describe no client outcome.

Seven records for one local effect
UTC timeObserved signalScenario decisionCumulative local effects
10:00:00evt_training_001 admitted; response lostDurable receipt, pending state; no business effect executed yet.0
10:00:01Worker A claims the eventAtomic work reservation; processing state.0
10:00:01Worker B sees the same eventDo not create a second processing owner or execute again.0
10:00:02Worker A commits the local transactionAtomically write initial activation and its completed state.1
10:00:03New delivery of evt_training_001Authenticate the new attempt, find the completed event and acknowledge without a second effect.1
10:00:04evt_training_002 targets the same initial activationRetain the distinct event; this business-operation key already exists.1
10:00:05evt_training_003 carries an earlier stateReject a regressive transition under the exercise policy and reconcile the source.1

The transaction described protects a local effect. It does not make an external service call atomic. After an unknown external result, retain uncertainty, query the source and apply the available idempotency mechanism before retrying. A transactional outbox can organise emission; it does not establish a universal exactly-once guarantee.

Ten denial and recovery tests to prepare

01 · Invalid signature: no effect

Present a delivery whose HMAC does not match the received body.

Expected: reject before admission and any effect. Retain a redacted reason and configuration version, without recording the key, token or sensitive body content.

02 · Reformatted JSON with the original signature

Replace A with B while keeping the signature computed for A. Both JSON strings decode to the same object.

Expected: verification failure. Capture the raw body at the right framework stage; do not try different reformattings to recover the signature.

03 · Correct signature, stale timestamp

In a Stripe test, authenticate a 09:50:00 timestamp while the acceptance-test clock is 10:00:00, with a configured five-minute tolerance.

Expected: reject freshness despite a correct HMAC. Separate message integrity, delivery age and event creation age.

04 · Same event, newly authenticated attempt

Receive two deliveries of the same event.id; the Stripe retry has its own timestamp and signature.

Expected: authenticate each attempt, then find durable state. A different signature does not justify a second business effect.

05 · Two events, one initial activation

Receive two event identifiers requesting the same fictional initial activation of case_42.

Expected: record both events and protect this operation with its business key. Do not apply this rule blindly to legitimate successive updates of the same object.

06 · Two workers at the same instant

Start two workers on the same pending item in the authorised test environment.

Expected: atomic acquisition and one effect transaction. “Read absent, then insert” without a constraint can allow both workers through; inspect business results, not logs alone.

07 · Queue unavailable before acknowledgement

Simulate queue or database failure before receipt has been durably confirmed.

Expected under our architecture: do not acknowledge as though the event were retained. Exercise the provider’s recovery strategy; GitHub does not automatically redeliver failures.

08 · Response lost after durable receipt

Drop the response after notification persistence; let processing continue under the exercise policy.

Expected: the next attempt finds pending, processing or completed state. The sender may not know receipt succeeded; network uncertainty must not duplicate the effect.

09 · Old event after a newer state

Deliver a notification with an earlier state after a newer notification.

Expected: apply permitted business transitions and reconcile current state when necessary. A fresh signing timestamp can carry an old event; it does not prove business-state novelty.

10 · Manual recovery overlaps automatic retry

Send manual recovery and Stripe automatic retry towards the same event.

Expected: both paths use the same concurrency and idempotency controls. A successful manual resend does not itself cancel automatic retries; verify closure with the provider.

Six useful operating records

  • Dated provider contract: environment, account, type, version, signature, timing and recovery.
  • Redacted admission trace: internal reference, verification result, clock and policy version.
  • Durable event state, work owner and evidence that the uniqueness constraint was tested.
  • Effect identity and legitimately repeatable operations; separate the event from the business object.
  • Failure queue with cause, attempt count, owner, next action and verified closure.
  • Source/application reconciliation over a defined period, including never-received notifications and uncertain external effects.

Provider contracts and primary sources

Sources reviewed on

API errors: types, statuses and evidence

An error response establishes neither failure nor safe replay

After interruption or an inconsistent response, a call may have produced an effect. Problem Details improves description; it does not prove no processing occurred or provide a universal retry policy.

  • Preserve HTTP status, received body and permitted operation identifier in controlled traces.
  • Check business state and idempotency rules before repeating an action.
  • Test public messages without exposing secrets, internal paths or account details.

Explore problem types and eight fictional responses →

IETF · M. Nottingham, E. Wilde, S. Dalal · RFC 9457 · July 2023 · obsoletes RFC 7807 · §3.1, §3.2, §4.2, §5

Official references

References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.