Inside a payment system, one event triggers many actions. A payment succeeds: the order is marked paid, stock is released for shipping, a receipt is emailed, the ledger is posted, a seller's balance goes up, an analytics event is sent. Each of those must happen once. Not zero times, or a customer who paid gets nothing; not twice, or a seller is credited double.
The difficulty is that every message between these steps can be lost or repeated, and every process can crash halfway. This guide explains how payment systems get exactly-once effects anyway, using techniques that are well established but easy to get subtly wrong. It's the internal counterpart to our guide on duplicate payments, which covers the customer-facing side.
Delivery guarantees, honestly stated
Message systems offer one of two practical guarantees, and you should know which you're relying on:
| Guarantee | What it means | Risk |
|---|---|---|
| At most once | Send and don't retry | Messages can be lost |
| At least once | Retry until acknowledged | Messages can be delivered more than once |
| Exactly once (effect) | At-least-once delivery plus processing that ignores repeats | Only as good as your idempotency |
For payments, losing messages is unacceptable, so almost everything uses at-least-once delivery: provider webhooks, message brokers, job queues. Exactly-once is then something you build, not something you buy: deliver at least once, and make every consumer idempotent.
The dual-write problem
The most common bug in event-driven payment code looks harmless. When a payment succeeds, the code updates the database and then publishes an event:
- Update the payment to succeeded and commit.
- Publish "payment.succeeded" to the message broker.
If the process crashes between the two steps, the database says paid but no event was ever sent: the order never ships. Reverse the order and a crash leaves an event for a payment the database doesn't show as paid. Wrapping both in a try-catch doesn't help, because the crash can happen anywhere. Two separate systems can't be updated atomically without a protocol that makes them one.
The transactional outbox
The standard fix is to write the event into the same database, in the same transaction as the state change, then have a separate relay publish it. The database transaction makes the state change and the event atomic: both happen, or neither does.
-- 1. State change and outbox row in one statement. The event is written
-- only if the payment actually moved to succeeded.
WITH changed AS (
UPDATE payments
SET status = 'succeeded', updated_at = now()
WHERE id = $1 AND status IN ('pending', 'unknown')
RETURNING id
)
INSERT INTO outbox (aggregate_id, type, payload)
SELECT id, 'payment.succeeded', $2 FROM changed;
-- 2. The relay claims unpublished rows. SKIP LOCKED lets several relay
-- workers run without picking up the same rows.
BEGIN;
SELECT id, type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;
-- ...publish each row to the broker, then mark them sent:
UPDATE outbox SET published_at = now() WHERE id = ANY($1);
COMMIT;The relay can still publish a row and crash before marking it sent, so it will publish that row again on restart. That's at-least-once delivery, and it's fine, because consumers are idempotent. Instead of polling, the relay can read the database's change log; change-data-capture tools such as Debezium provide an outbox router for exactly this.
If event order matters, as it usually does per payment, note that several relay workers with SKIP LOCKED can publish rows out of order. Either run one relay per partition of payments, or make consumers tolerate reordering, as below.
The idempotent consumer
Every consumer must be able to receive the same message twice and act once. The reliable way is to record each message ID as processed in the same transaction as the consumer's own changes:
async function onPaymentSucceeded(message: PaymentEvent): Promise<void> {
await db.transaction(async (tx) => {
const fresh = await tx.query(
"INSERT INTO processed_messages (consumer, message_id) VALUES ('fulfilment', $1) " +
"ON CONFLICT DO NOTHING RETURNING message_id",
[message.id],
);
if (fresh.rowCount === 0) return; // seen before: do nothing, then acknowledge
await tx.query(
"UPDATE orders SET status = 'paid' WHERE id = $1 AND status = 'awaiting_payment'",
[message.orderId],
);
// Further events go through this service's own outbox, in the same transaction.
await tx.query(
"INSERT INTO outbox (aggregate_id, type, payload) VALUES ($1, 'order.paid', $2)",
[message.orderId, JSON.stringify({ orderId: message.orderId })],
);
});
// Acknowledge to the broker only after the transaction has committed.
}Two details matter. The processed-message record and the business change commit together, so a crash can't leave one without the other. And the business change is itself conditional ("paid only if awaiting payment"), which protects you even if a message arrives under a different ID, as can happen when the same fact is reported by both a webhook and a status check.
Side effects outside your database
Emails, provider API calls and other external effects can't join your database transaction. Make them idempotent another way:
- For provider calls such as refunds and payouts, derive the idempotency key from something stable, such as the refund ID, so a repeated message sends the same key and the provider returns the original result.
- For emails and notifications, send them from an outbox row created with the state change, and record the send against that row, so a duplicate message doesn't create a second email.
- For anything that can't be made idempotent, accept that it may very occasionally repeat, and make sure it's harmless if it does.
Incoming webhooks: the inbox
Provider webhooks are at-least-once deliveries from outside, and the same patterns apply in reverse. Verify the signature, store the raw event with the provider's event ID under a unique constraint, acknowledge, and process from that stored "inbox" asynchronously. A duplicate delivery fails the unique insert and is acknowledged without being processed again. Our guide to payment gateway webhooks covers verification and acknowledgement.
Ordering
At-least-once delivery with retries means events can arrive out of order. There are two approaches, and payment systems usually need both:
- Keep order where it matters: route all events for one payment through the same partition or queue, so they're processed one at a time and in sequence.
- Tolerate disorder anyway: apply events through a state machine that refuses backward moves (a succeeded payment never returns to pending), or carry a version number and ignore events older than what you've stored.
The second approach is what saves you when the first isn't perfect, for example when a provider's webhook and your own status check report the same payment in different orders.
What about "exactly-once" message brokers?
Some brokers offer exactly-once features. Apache Kafka, for example, documents exactly-once semantics for reading from and writing to Kafka within a transaction. That's valuable for stream processing that stays inside Kafka, but it doesn't extend to your database, a provider's API or an email gateway. As soon as a consumer changes something outside the broker, you're back to at-least-once plus idempotency.
Testing that it actually works
These designs fail only under conditions that normal testing never produces, so produce them deliberately:
- Kill the process at every step: after the state change, after publishing, before acknowledging. Check that nothing is lost and nothing happens twice.
- Deliver every message twice in a test environment, and some of them three times.
- Shuffle message order within a payment and check the final state is right.
- Replay a day of production events into a staging copy and compare the results with production.
- Reconcile: count orders paid, ledger postings and provider settlements for the same period. They should agree exactly. Our guide to double-entry ledgers shows how a ledger makes this check precise.
A checklist
For each event-driven flow in your payment system:
- State changes and the events they cause are written in one database transaction (outbox).
- Every consumer records processed message IDs in the same transaction as its work.
- Every state change is conditional on the current state.
- External calls use stable idempotency keys.
- Incoming webhooks are stored with a unique event ID before processing.
- Events for one payment are processed in order, and out-of-order events are ignored safely.
- Crash, duplicate and reordering tests run regularly.
If you'd rather receive one clean event stream than build this yourself, LILPE's unified webhooks page describes how provider callbacks are verified before being passed on, and notes that delivery order isn't guaranteed, which is exactly why your consumers still need the patterns above.
Frequently asked questions
Is exactly-once delivery possible?
Not as a guarantee from the network: a sender can't always know whether a message arrived, so it either risks losing it or sends it again. Systems achieve exactly-once effects instead, by delivering at least once and making processing idempotent so repeats change nothing.
What is the transactional outbox pattern?
A way to change state and publish an event atomically. The event is written to an outbox table in the same database transaction as the state change, and a separate relay publishes outbox rows to the message broker. If the transaction commits, the event will be published; if it rolls back, the event never existed.
What is an idempotent consumer?
A message consumer that produces the same result whether it receives a message once or several times. Typically it records each processed message ID in the same transaction as its changes and skips messages it has already recorded.
How do I handle duplicate payment webhooks?
Store each webhook with the provider's event ID under a unique constraint before processing, skip IDs you've already stored, and make the resulting state changes conditional, such as marking an order paid only if it isn't already.
Official sources
- Chris Richardson, microservices.io — Pattern: Transactional outbox
- Debezium documentation — Outbox event router
- Apache Kafka documentation — Message delivery semanticsAt-most-once, at-least-once and exactly-once semantics, and transactional writes within Kafka.
- PostgreSQL documentation — SELECT, the locking clause (FOR UPDATE, SKIP LOCKED)
Last reviewed . Examples, amounts and screens marked illustrative are not LILPE figures.
