Every payment provider has bad days. A data centre has trouble, a bank behind the provider slows down, a certificate expires, a release goes wrong. For the merchant, the symptom is the same: customers can't pay, and the order count drops in real time. What separates a contained incident from an expensive one is mostly what happens in the first hour, and whether anything is left inconsistent afterwards.
This runbook is written for merchants and the engineers who support them. It assumes nothing about your provider or stack. Adapt it, write your own names and thresholds into it, and rehearse it before you need it.
How you find out
The best case is that your monitoring tells you before your customers do. The signals that move first are usually technical, not commercial:
- Timeouts and error responses from one provider rising sharply.
- Payments piling up in pending, and the age of the oldest pending payment growing.
- Success rate falling on one provider or one method, compared with the same hour last week.
- Webhooks arriving late or not at all.
- Support tickets and social posts saying "I paid but it says failed".
Don't wait for the provider's status page. Status pages are useful for confirmation, but they're often updated after merchants notice. Our guide to payment success rate explains how to alert on these signals without false alarms.
The first hour at a glance
Times are a guide to the order of work, not targets to hit.
- 0–5 minAlert fires. Someone takes charge of the incident and opens a channel.
- 5–15 minConfirm and scope: is it you or the provider? Which methods, banks and providers?
- 15–25 minDecide: route elsewhere, disable the affected method, or show a notice. Stop retry storms.
- 25–40 minCommunicate: checkout message, support script, provider ticket with evidence.
- 40–60 minHold steady: watch recovery signals, keep in-flight payments unknown, not failed.
- AfterRestore gradually, reconcile the outage window, follow up with customers, write the review.
Step 1: confirm it isn't you
A surprising share of "provider outages" turn out to be on the merchant's side. Check these quickly, because the fix is in your hands:
| Symptom | Likely cause | Check |
|---|---|---|
| Every request rejected with an authentication error | Rotated or expired API key, wrong environment | Recent config changes; test-versus-live keys |
| Connection or TLS errors to the provider | Expired certificate, DNS change, firewall or proxy rule | Your outbound network and certificates |
| Failures began exactly at a deploy | Your release | Roll back first, investigate second |
| One method failing everywhere, others fine | That method's rail or a shared bank | Same method on other providers, if you have them |
| One bank failing on every provider | The bank | Bank-level breakdown; nothing to fix on your side |
| Everything on one provider failing | The provider | Provider status page and account manager |
Step 2: contain the impact
Once you know where the problem is, choose the least disruptive option that stops customers failing:
- Route new payments to another provider for the affected methods, if you have one connected and approved for that traffic.
- Disable only the affected method or flow at checkout, and keep the rest open. Customers can usually pay another way.
- If nothing works, show a clear notice at checkout rather than letting customers fail repeatedly, and keep orders open so they can pay later.
- Pause automatic retries to the failing provider, or slow them down sharply. Retrying into an outage adds load to a provider trying to recover and can trip its rate limits.
- Hold outgoing operations that depend on the provider, such as refunds and payouts, rather than retrying them in a loop. Queue them for after recovery.
Step 3: don't create a second problem
Outages produce the conditions for duplicates and lost payments: timeouts, uncertain outcomes and customers trying again. Protect yourself:
- Treat timed-out payments as unknown, not failed. Payments in flight when the provider stalled may still complete. Our guide to payment API timeouts explains how to resolve them.
- Don't move a pending or unknown payment to another provider. Only new payments should be routed elsewhere.
- Keep customers from paying twice: reuse their open order, and tell them on screen if an earlier attempt is still being confirmed. Our guide to duplicate payments covers the safeguards.
- Keep accepting webhooks. If your own endpoint is struggling, queue events rather than rejecting them, so the provider's retries can catch up later.
Step 4: communicate
Three audiences need to hear from you, and each needs something different:
- Your team: one channel, one person in charge, regular short updates. The Google SRE book's guidance on managing incidents (clear roles, a single coordinator, a living incident document) is a good model for any size of team.
- Customers: a plain message at checkout ("Card payments are temporarily unavailable; please use UPI") and a script for support. Tell anyone whose money was debited that failed payments are reversed and give them a reference to quote.
- The provider: open a ticket with evidence, not just "payments are failing". Include the time it started, affected methods, error codes, sample payment IDs with timestamps, and your account identifier. Good evidence gets faster answers.
Step 5: recover carefully
When the provider reports a fix, don't send all traffic back at once. Restore a small share first, watch success rate, errors and response times for that share, then step up. A provider that has just recovered can fall over again under a sudden full load, and a second outage is worse than a slow return.
Release queued refunds and payouts in batches, watching for errors, and check each for duplicates before it goes: an operation that timed out during the outage may already have gone through.
After the incident: reconcile the window
The incident isn't over when payments work again. Payments started during the outage need to end up in exactly the right state:
- List every attempt created during the outage window, plus a margin either side.
- For each one still pending or unknown, fetch its final status from the provider.
- Mark paid any order whose payment succeeded but which your system shows as unpaid, and fulfil it.
- Refund any duplicates: orders with more than one successful payment.
- Confirm that failed payments where customers were debited are being reversed, and answer those customers.
- Match the window against the provider's transaction and settlement reports when they arrive. Our reconciliation guide explains the matching.
Write the review, then fix the runbook
Hold a short, blameless review within a few days: what happened, when you knew, what you did, what took longest, and what you'll change. The aim is better systems, not finding someone to blame, which is also the spirit of the SRE book's chapter on postmortem culture. Typical follow-ups are an alert that should have fired earlier, a switch that needed a deploy, or a dependency on a single provider for a method customers rely on.
Prepare before the next one
Most of what makes an outage cheap is done in advance:
- A written runbook with names, contacts and the switches above, stored somewhere that doesn't depend on the systems that might be down.
- Alerts on errors, timeouts, pending age and success rate per provider and method.
- Feature flags to disable a method, and routing rules to move traffic, that anyone on call can use.
- A second provider approved for your most important methods, if your volume justifies it, tested with real traffic before you need it.
- Rehearsals: run the runbook once against a planned maintenance window or a test environment.
If you're weighing a second provider, LILPE's automatic failover page describes how it measures provider health per method and pauses an unhealthy provider, and its multiple payment providers page describes running providers side by side.
Frequently asked questions
How do I know if my payment gateway is down?
Watch your own signals rather than waiting for the provider's status page: a jump in timeouts and errors from one provider, payments stuck in pending, and a fall in success rate on that provider compared with the same hour last week. Then rule out your own causes, such as an expired key or a recent deploy.
What should I do if my payment gateway is down?
Confirm the cause, then contain it: route new payments to another provider if you have one, or disable just the affected method and keep the others open. Treat in-flight payments as unknown rather than failed, tell customers what's happening, open a ticket with evidence, and reconcile the outage window afterwards.
Will customers lose money during a payment gateway outage?
Money debited for a payment that ultimately fails is reversed under RBI's turnaround-time framework for failed transactions, with compensation for delays beyond the set timelines. The bigger risk is customers paying twice or paying for orders your system shows as unpaid, which is why the clean-up after the outage matters.
Should I retry payments on another gateway during an outage?
Route new payments elsewhere, yes, if you have another provider set up. Don't move payments that are already pending or unknown on the failing provider: they may still succeed, and moving them can charge the customer twice.
How can I reduce the impact of payment gateway downtime?
Monitor per provider and method, keep switches to disable methods or move traffic that don't need a deploy, connect a second provider for your most important methods, and rehearse the runbook. Most of the cost of an outage is decided by what was prepared before it.
Go deeper
Official sources
- Google — Site Reliability Engineering, chapter 14: Managing IncidentsSeparating responsibilities, a recognised command post, a live incident document and clear handoffs.
- Google — Site Reliability Engineering, chapter 15: Postmortem Culture: Learning from FailureBlameless postmortems focused on causes and follow-up actions.
- RBI — Harmonisation of Turn Around Time (TAT) and customer compensation for failed transactions (20 Sep 2019)Timelines for reversing failed transactions by channel, with compensation for delays.
Last reviewed . Examples, amounts and screens marked illustrative are not LILPE figures.
