Ship the payments retry queue
A durable retry queue replaces inline payment-capture retries, demonstrated here as a deck of Part acts with a TableOfContents overview.
Part 1Context
1.1 / Status quo
Inline retries delay checkout
- Inline retries couple checkout latency to processor health.
- Retry state lives in process memory, so every deploy loses it.
- Operators cannot see, pause, or force a retry.
1.2 / Success looks like
Success looks like
- A failed capture is retried on schedule even across API-server restarts.
- Operators watch every attempt in one audit trail.
Part 2The proposal
2.1 / The retry queue
The retry queue
A queue worker owns retries end to end.
2.1.1 / The worker
What the queue worker does on every attempt.
- Claims due schedules with explicit state per attempt.
- Persists an idempotency key for each logical capture before its first processor call and reuses that key for every retry.
- Reconciles in-flight or unknown attempts with the processor before replay; a confirmed capture is recorded as complete, while an unresolved outcome stays pending.
- Applies bounded backoff, so a stuck capture never retries forever.
2.1.2 / The audit trail
Where every attempt becomes visible to operators.
- Every state change lands in the audit trail as it happens.
- Operators pause, force, or cancel from the same view.
Part 3Shipping & your review
3.1 / Sequencing
Sequencing
We need to agree on the landing order before the schema ships.
- Land the schema migration first, behind no user-facing change.
- Ship the queue worker second, shadowing inline retries.
- Cut over and add operator controls last.
3.2 / Acceptance criteria
Restarts preserve scheduled retries
- A failed capture is retried on schedule after an API-server restart.
- If the processor captures successfully and the API server restarts before recording the result, recovery reconciles that capture without issuing a duplicate.
- Operators can pause, force, and cancel retries per merchant.