Webhook Delivery Status Runbook

Use this page for BDT-319 on the M6 Observability + Settlement Ops milestone.

The goal is to make failed merchant callbacks visible and recoverable without exposing secrets or creating an unbounded retry loop.

Delivery attempt record

One delivery_id groups all attempts for a single outbound webhook event. Each attempt should expose the same operational fields so support can tell whether a delivery is still moving, retrying, or terminal.

Field	Meaning	Merchant-safe
`delivery_id`	Stable identifier for the outbound webhook event	Yes
`attempt_id`	Unique identifier for one send attempt	Yes
`attempt_number`	1-based attempt count for the delivery	Yes
`status`	`queued`, `sending`, `delivered`, `retry_scheduled`, `failed`, or `undeliverable`	Yes
`event_type`	Webhook event name such as `payment_session.completed`	Yes
`merchant_id` / `merchant_name`	Target merchant context	Yes
`webhook_url`	Destination endpoint URL, without credentials	Yes
`response_code`	Exact HTTP status if the merchant replied, or a transport class such as `timeout` / `dns_error` / `tls_error`	Yes, redacted to code or class only
`latency_ms`	Time from dispatch to response or failure	Yes
`next_retry_at`	Next scheduled automatic retry time	Yes
`retry_count`	Automatic attempts made so far	Yes
`manual_retry_count`	Manual retries made so far	Yes
`last_error_class`	Normalized failure class for dashboards and support	Yes
`request_id` / `session_id`	Correlation IDs	Yes
`payload_hash`	Digest for correlation without storing the raw body	Yes
`created_at` / `updated_at` / `last_attempt_at`	Operational timestamps	Yes

Keep the raw response body, response headers, delivery secret, signature material, and operator notes out of the merchant surface. Store them only in operator audit or incident records.

Status and retry meaning

Latest result	Status	Next retry	Notes
`2xx` response	`delivered`	None	Delivery succeeded. Show the final latency only.
`timeout` / transport error	`retry_scheduled`	Compute from backoff	Treat as transient unless it repeats across several attempts.
`5xx` response	`retry_scheduled`	Compute from backoff	Merchant endpoint is unhealthy or overloaded.
`429` response	`retry_scheduled`	Compute from backoff	Merchant is rate limiting the callback path.
`4xx` other than `429`	`retry_scheduled` or `failed`, depending on retry budget	Usually present until the retry budget is exhausted	Usually a payload, auth, or callback configuration problem.
Retry budget exhausted	`undeliverable`	None	Do not keep retrying automatically. Escalate to support and the merchant.

For the detail page and dashboard, show the last attempt's response_code, latency_ms, retry_count, and next_retry_at together. That combination tells operators whether the delivery is healthy, backing off, or stuck.

Bounded manual retry policy

Manual retry is a recovery action, not a new delivery primitive.

Only an authorized operator can trigger a manual retry.
Manual retry is allowed only for failed or undeliverable deliveries.
The retry must reuse the original payload and webhook identity. It creates a new attempt record but does not mutate the original event.
Cap manual retries at 3 per delivery_id within 24 hours.
A fourth retry requires incident-owner approval and a written reason in the audit trail.
Do not use manual retry while a healthy automatic retry is already scheduled or while the callback endpoint is still failing with the same class of error.
Stop manual retries once the failure pattern points to a merchant-side fix that has not been applied.

Recommended manual retry reasons:

endpoint_fixed
secret_rotated
transient_outage
support_replay

Audit requirements

Every manual retry must produce an immutable audit event.

Record:

delivery_id
attempt_id
operator_id
operator_role
trigger_source such as dashboard, admin tool, or support workflow
reason
incident_id or ticket reference when available
before_status
after_status
manual_retry_count_before
manual_retry_count_after
response_code
latency_ms
created_at

Also retain the raw transport details needed for forensic review, but only in operator-only storage:

response body hash, not the raw body
response headers, redacted as needed
callback URL at the time of the attempt
request and trace identifiers
approval history when a retry exceeds the normal cap

Merchant-safe visibility

Merchant-facing views should explain what happened without exposing internal support actions.

Show:

delivery status
attempt count
response code or response class
latency
next retry ETA
a short failure class such as timeout, 5xx, auth, or configuration

Hide:

operator identity
internal notes
raw response bodies
response headers
webhook secrets
signature values
IP addresses and internal hostnames

If the merchant needs more detail, provide a support-safe summary that names the failure class and the next action, not the raw transport data.

Failure handling

Failure class	Default handling	Manual retry guidance
`timeout`, `dns_error`, `tls_error`, connection reset	Keep automatic retries with backoff	Reasonable after the merchant confirms the endpoint is reachable
`5xx`	Keep automatic retries with backoff	Reasonable after the merchant says the service is healthy again
`429`	Keep automatic retries with backoff	Only after the merchant removes the rate limit or increases capacity
`401` / `403`	Treat as an auth or signature problem	Do not keep retrying until the merchant updates the webhook secret or verification logic
`404` / `410`	Treat as a stale callback URL	Stop retries and ask the merchant to update the endpoint
Persistent `4xx` payload rejection	Treat as a contract or configuration problem	Fix the payload or merchant handler before retrying again

If the same delivery fails repeatedly with the same class of error, stop the retry loop, mark the delivery undeliverable, and surface the issue to support and the merchant.

Webhook Delivery Status Runbook

On this page