Houston, We Have a Bank Outage

Stéphane Derosiaux November 17, 2026 9 min read
Wireframe line art on a dark teal field: a large classical bank on the left sends two streams of data cubes toward two smaller banks on the right; both streams snap in the middle under a soft lime glow, and the cubes past the break tumble down into the dark.

You've probably seen the news. After a string of recent outages, Monday was the turn of several large US banks. Customers couldn't log in, move money or, at some branches, deposit a check, and Downdetector reports for one bank jumped from about 5,000 to more than 20,000 in under an hour.

It started with "some transactions are taking longer than usual to complete", then software reality kicked in: "a software bug in on-premise hardware systems".

Between 2023 and 2025, the largest UK banks reported at least 803 hours of unplanned outages (UK Treasury Committee). There is definitely a systemic issue somewhere, and it's probably infrastructure resiliency.

In many banks, Kafka is the real-time nervous system of the business: payments, fraud checks, account notifications. When it fails, customer reports come in instantly.

"If we have any issues on log on, [the press] know about it straight away and it's on the news."

Head of streaming platform at a bank

What follows an outage is probably the biggest sink of money and time, because everything has to be reconciled:

WHAT IT LEAVES BEHINDoutageservice returnslate fees found and refundedduplicates matched and reversedledger reconciled with networks and other bankssupervisory reviewfines, sometimes years laterhoursdaysweeksyears
Service comes back in hours; the work it leaves runs much longer.

DR plans are often written for the day a data center goes dark or a cloud region fails, and traffic has to move to the other one. Surprisingly, the top causes of the major incidents UK retail banks reported to the FCA in 2025 were change management (24%) and third-party failures (21%). The bank's statement points at three other failures:

What the bank saidWhat may happenWhat the platform needs
"a software bug in on-premise hardware systems"A software or data fault can sit on the secondary tooA way to contain it where it enters
"those banks may have been impacted"Trouble at one bank crosses into its counterpartiesIsolation per counterparty flow
"taking longer than usual to complete"Transactions wait in queues until the fixA controlled drain
If you work at a bank, you probably have many more examples. We'd love to hear them, and to help.
What does this post change for your team?ClaudeChatGPT

The Reference Architecture

Our team works with platform teams to design resilient Kafka architectures. Together, we work through:

  • the communication flows
  • what the architecture must survive
  • what the network, security and platform teams need to build it, from DNS to load balancers, firewalls and certificates
  • how to test and replay failures step by step
  • the runbooks to scale, upgrade, fail over and fail back

We do it with customers in an architecture blueprint tool we built for the job. Here it holds a region-loss design across AWS and an on-premises data center, with Conduktor Gateway routing the traffic and MirrorMaker 2 replicating the data:

The Gateway Architecture Blueprint showing a region-loss design: an on-premises data center and two cloud regions with their networks, load balancers, an active Gateway and a standby Gateway, two Kafka clusters, MirrorMaker 2 and a stopped failback copy, with output tabs for Deploy, Network, Infra security, Kafka security, Operate and Client
The architecture blueprint our customer engineers use, here on a region-loss design. Click to enlarge.

Switching Kafka connections without the applications noticing becomes the platform team's job. Application recovery and ledger integrity still need their own owners.

Contain or Switch: No Magic Wand

Replication tools such as MirrorMaker 2 and Cluster Linking copy every record to the secondary, including a malformed payment message or the output of a bad release. If the fault is in the data or in an application, failing over moves you to a second copy of the problem. It doesn't matter how many data centers you have if you run the same software and data in all of them.

So the first job is to find where the fault is: in the infrastructure or in the data. You really don't want it to travel into your safety net. Someone has to own the decision to fail over, and it's often unclear who that person is.

"We could have just failed over and call it a day, but I wasn't sure who was going to make that decision."

Platform engineer at an asset manager

This incident can follow two paths:

Incident callone named owner decidesWhere is the faultin the data or the codein the site or the clusterCONTAINMENTFAILOVERContain at Gatewaydata quality rules block new bad recordsKeep other flows runningfraud, ledger, notifications keep readingRepair what replicatedplan written before the incidentFence the old primary firstno writer reaches it, even in a partial partitionSwitch at Gatewayone config change, same addressConsumers resumefrom offsets synced beforehandFail back laterreverse copy, reconcile, switch back

When the secondary is clean, failing over is possible, but there is no magic wand. Replication between two clusters is asynchronous: the primary acknowledges a payment before MirrorMaker 2 or Cluster Linking copies it, so the secondary always runs behind, by seconds or more under load. When a site dies for real, the payments acknowledged but not yet copied are stuck there until it comes back, or gone with it.

Only a stretch cluster (one Kafka cluster spread across sites) copies every payment to the other site before acknowledging it. The price is latency on every write: about 15 ms between nearby US regions, 60 to 70 ms coast to coast, about 80 ms from the US East Coast to London. With two clusters, your RPO (how much data you can lose) is the replication lag at the moment of failure.

Two buttons meme: a hand hovers between a button labelled Fail over now, lose the last payments and a button labelled Wait, extend the outage, above a sweating man labelled The platform team on the incident call

Then the switch itself takes time. Without a single entry point like Conduktor Gateway, every application team has to update its connection settings and redeploy. One bank told us its Kafka failover takes two hours, with business stopped; another runs about 200 applications on Kafka (good luck synchronizing every team). The hard part is the coordination.

With Gateway, it is one switch for every application, and its chaos interceptors let teams check in pre-production that their apps survive it.

Data makes it harder: producers and consumers both have to move, in the right order, for operations to continue safely. Consumers are the tricky part:

  • Offsets. MirrorMaker 2 checkpoints translated consumer offsets every 60s and writes them into the secondary's consumer groups. Without that, or another tool that translates offsets, a group falls back to auto.offset.reset: latest skips the backlog and earliest replays everything retained. Not what you want.
  • Topic names. MirrorMaker 2 prefixes replicated topics with the source cluster's name by default, so consumers fail to find them unless they are reconfigured, on failover and again on failback.

If your provider offers managed replication (Confluent Cluster Linking, MSK Replicator, Redpanda Shadowing), use it: it handles offsets and topic names for you and is less to run. It is still asynchronous, so it shrinks the RPO without removing it.

Dependencies between banks

Banks work together, and payments cross payment networks. When one bank fails, payments to and from it get stuck. That's why, during Monday's outage, other banks said their own systems were fine while their customers' transfers to and from the failing bank were still delayed. Payment networks can do the same: in November 2023, one bad file at a US payment network held up about 900,000 deposits across several large banks.

Kafka platforms increasingly sit in the middle of these flows: partners read from them, and they read from partners. When a partner stops answering, three things matter:

THEY READ FROM YOUYOUR PLATFORMYOU READ FROM THEMpayment networksaggregatorspartnersKafkatopics · retentionquotas per clientowners per flow123payment networksprocessorsother banksstops answering1work waits in Kafka while retention outlasts the outage and the drain2a quota per partner keeps one catch-up from starving the rest3a named owner per flow, reachable within minutes

In our reference design, Gateway partner zones give each external partner its own access to shared topics, and Console shows who reads each topic and who owns each application.

Draining the Backlog and self-DDoSing

When transactions are "taking longer than usual", they are waiting somewhere. Once the fix lands, consumers pull that backlog at their own pace and load the ledger database, the fraud model, partner APIs and the Kafka brokers.

This is where you can self-DDoS your own organization. One bank paused its message consumers by mistake, then resumed them all at once. An engineer wrote in the postmortem: "We had distributed millions of messages into our own services, effectively a self-DDoS." The fix was to resume one channel at a time, so the system and everything behind it could absorb the load gradually.

Drain no faster than your slowest downstream system can handle:

outage startsfix landsbacklog cleared (paced)timeload on the ledger and fraud modelwhat the slowest downstream system can absorbnormal loadbacklog builds up in Kafkaevery consumer at oncesecond incidentpaced drainledger and fraud model down againdrain time ≈backlog ÷ (sustainable rate − new arrivals)
Illustrative: a paced drain stays under what downstream systems can absorb and finishes; an unpaced one becomes a second incident.

Kafka's consumer quotas and per-application concurrency let the platform team release the backlog flow by flow. You still have to make sure no payment is lost or processed twice:

  • Duplicate payments. Kafka only removes duplicates caused by a producer's own retries. To avoid posting a payment twice, the ledger checks an idempotency key, such as the payment ID, before posting.
  • Lost payments. When a producer can't deliver to Kafka in time (two minutes by default), it gives up and fails. A good pattern is the outbox: the application writes the payment to an outbox table in its own database, in the same transaction as its business update, and a separate process, often change data capture (CDC) with Debezium, sends it to Kafka once Kafka is reachable again.
  • Catch-up order. Decide before the incident which systems catch up first, for instance payments and fraud checks before internal reporting.

In our reference architecture, Gateway rate limits pace the applications that replay writes, and its produce policies enforce acks=all and idempotence, so a misconfigured producer can't lose acknowledged writes or duplicate them on retry.

The Architecture to Survive a Region Loss

The blueprint has one design per failure:

  • Losing Kafka: Gateway switches to the DR cluster.
  • Losing the Gateway deployment: a standby Gateway on the same Kafka takes over.
  • Losing a whole region: the design below, across two cloud regions and the bank's own data center.
ON-PREM DATA CENTERgateway.mycompany.com → F5 BIG-IP pairREGION A · ACTIVEgateway.mycompany.com → Gateway AREGION B · STANDBYgateway.mycompany.com → Gateway BVPNVPNpeeringerp-appF5 BIG-IP pairpayments-appNLBGateway A3 instancesKafka primary3 brokers · fenced at failoverpayments-appstopped copyNLBGateway Bstandby · 0 instancesMirrorMaker 2copies data + offsetsKafka secondary3 brokersGateway state replicated(accounts, interceptors)failback (stopped)
  • Same address everywhere. gateway.mycompany.com resolves to Gateway A in Region A, to the standby Gateway in Region B, and to the F5 BIG-IP pair in the data center. Applications never change their bootstrap address.
  • A standby that starts on demand. The standby Gateway runs no instance until a region is lost. Then the runbook cuts off the old region, checks that Gateway's configuration (service accounts, interceptors) is on the secondary, and starts the standby and the payments-app copy. Keeping it stopped prevents writes to both clusters at once and avoids paying for cross-region traffic. If that cold start is too slow for your recovery target, keep the standby running.

The blueprint generates that runbook from the design itself: one script with read-only checks, confirmed steps for each failover, fence and failback, and a drill for pre-production.

The failover runbook section of a blueprint report: one generated script whose commands check which failure happened, fail over the Gateway, Kafka or the whole region, verify, fence the old side and fail back, each marked either writes only probe records or changes things, asks you to confirm
Part of the failover runbook the blueprint generates from the design.

Choosing Your Trade-offs

Making a bank's Kafka platform resilient is hard: no single team owns the whole path of a payment. The fault can sit in the data, the code, a site or a counterparty, replication always leaves a gap, and recovery depends on ledgers and partners that Kafka doesn't control. No tool makes an outage disappear.

So choose your trade-offs with the business, before the incident:

ChoiceYou gainYou pay
Stretch clusterNo data loss when a site diesLatency on every write
Two clustersFast writesSeconds of payments at risk
Warm standbyFast recoveryMoney every day
Cold standbyLower costMinutes during the outage
Then agree on:
  • How much data you can lose: your RPO.
  • How long you can be down: your RTO (recovery time).
  • Who decides on the incident call.

To work through these trade-offs on your platform, book 45 minutes with our engineers: we'll map one of your flows in the blueprint, and you leave with its design and open issues.

Book a Kafka resilience review

Related: Kafka Disaster Recovery Beyond Replication