You've probably seen the news. After a string of recent outages, Monday was the turn of several large US banks. Customers couldn't log in, move money or, at some branches, deposit a check, and Downdetector reports for one bank jumped from about 5,000 to more than 20,000 in under an hour.
It started with "some transactions are taking longer than usual to complete", then software reality kicked in: "a software bug in on-premise hardware systems".
Between 2023 and 2025, the largest UK banks reported at least 803 hours of unplanned outages (UK Treasury Committee). There is definitely a systemic issue somewhere, and it's probably infrastructure resiliency.
In many banks, Kafka is the real-time nervous system of the business: payments, fraud checks, account notifications. When it fails, customer reports come in instantly.
"If we have any issues on log on, [the press] know about it straight away and it's on the news."
Head of streaming platform at a bank
What follows an outage is probably the biggest sink of money and time, because everything has to be reconciled:
DR plans are often written for the day a data center goes dark or a cloud region fails, and traffic has to move to the other one. Surprisingly, the top causes of the major incidents UK retail banks reported to the FCA in 2025 were change management (24%) and third-party failures (21%). The bank's statement points at three other failures:
| What the bank said | What may happen | What the platform needs |
|---|---|---|
| "a software bug in on-premise hardware systems" | A software or data fault can sit on the secondary too | A way to contain it where it enters |
| "those banks may have been impacted" | Trouble at one bank crosses into its counterparties | Isolation per counterparty flow |
| "taking longer than usual to complete" | Transactions wait in queues until the fix | A controlled drain |
The Reference Architecture
Our team works with platform teams to design resilient Kafka architectures. Together, we work through:
- the communication flows
- what the architecture must survive
- what the network, security and platform teams need to build it, from DNS to load balancers, firewalls and certificates
- how to test and replay failures step by step
- the runbooks to scale, upgrade, fail over and fail back
We do it with customers in an architecture blueprint tool we built for the job. Here it holds a region-loss design across AWS and an on-premises data center, with Conduktor Gateway routing the traffic and MirrorMaker 2 replicating the data:

Switching Kafka connections without the applications noticing becomes the platform team's job. Application recovery and ledger integrity still need their own owners.
Contain or Switch: No Magic Wand
Replication tools such as MirrorMaker 2 and Cluster Linking copy every record to the secondary, including a malformed payment message or the output of a bad release. If the fault is in the data or in an application, failing over moves you to a second copy of the problem. It doesn't matter how many data centers you have if you run the same software and data in all of them.
So the first job is to find where the fault is: in the infrastructure or in the data. You really don't want it to travel into your safety net. Someone has to own the decision to fail over, and it's often unclear who that person is.
"We could have just failed over and call it a day, but I wasn't sure who was going to make that decision."
Platform engineer at an asset manager
This incident can follow two paths:
When the secondary is clean, failing over is possible, but there is no magic wand. Replication between two clusters is asynchronous: the primary acknowledges a payment before MirrorMaker 2 or Cluster Linking copies it, so the secondary always runs behind, by seconds or more under load. When a site dies for real, the payments acknowledged but not yet copied are stuck there until it comes back, or gone with it.
Only a stretch cluster (one Kafka cluster spread across sites) copies every payment to the other site before acknowledging it. The price is latency on every write: about 15 ms between nearby US regions, 60 to 70 ms coast to coast, about 80 ms from the US East Coast to London. With two clusters, your RPO (how much data you can lose) is the replication lag at the moment of failure.

Then the switch itself takes time. Without a single entry point like Conduktor Gateway, every application team has to update its connection settings and redeploy. One bank told us its Kafka failover takes two hours, with business stopped; another runs about 200 applications on Kafka (good luck synchronizing every team). The hard part is the coordination.
With Gateway, it is one switch for every application, and its chaos interceptors let teams check in pre-production that their apps survive it.
Data makes it harder: producers and consumers both have to move, in the right order, for operations to continue safely. Consumers are the tricky part:
- Offsets. MirrorMaker 2 checkpoints translated consumer offsets every 60s and writes them into the secondary's consumer groups. Without that, or another tool that translates offsets, a group falls back to
auto.offset.reset:latestskips the backlog andearliestreplays everything retained. Not what you want. - Topic names. MirrorMaker 2 prefixes replicated topics with the source cluster's name by default, so consumers fail to find them unless they are reconfigured, on failover and again on failback.
If your provider offers managed replication (Confluent Cluster Linking, MSK Replicator, Redpanda Shadowing), use it: it handles offsets and topic names for you and is less to run. It is still asynchronous, so it shrinks the RPO without removing it.
Dependencies between banks
Banks work together, and payments cross payment networks. When one bank fails, payments to and from it get stuck. That's why, during Monday's outage, other banks said their own systems were fine while their customers' transfers to and from the failing bank were still delayed. Payment networks can do the same: in November 2023, one bad file at a US payment network held up about 900,000 deposits across several large banks.
Kafka platforms increasingly sit in the middle of these flows: partners read from them, and they read from partners. When a partner stops answering, three things matter:
In our reference design, Gateway partner zones give each external partner its own access to shared topics, and Console shows who reads each topic and who owns each application.
Draining the Backlog and self-DDoSing
When transactions are "taking longer than usual", they are waiting somewhere. Once the fix lands, consumers pull that backlog at their own pace and load the ledger database, the fraud model, partner APIs and the Kafka brokers.
This is where you can self-DDoS your own organization. One bank paused its message consumers by mistake, then resumed them all at once. An engineer wrote in the postmortem: "We had distributed millions of messages into our own services, effectively a self-DDoS." The fix was to resume one channel at a time, so the system and everything behind it could absorb the load gradually.
Drain no faster than your slowest downstream system can handle:
Kafka's consumer quotas and per-application concurrency let the platform team release the backlog flow by flow. You still have to make sure no payment is lost or processed twice:
- Duplicate payments. Kafka only removes duplicates caused by a producer's own retries. To avoid posting a payment twice, the ledger checks an idempotency key, such as the payment ID, before posting.
- Lost payments. When a producer can't deliver to Kafka in time (two minutes by default), it gives up and fails. A good pattern is the outbox: the application writes the payment to an outbox table in its own database, in the same transaction as its business update, and a separate process, often change data capture (CDC) with Debezium, sends it to Kafka once Kafka is reachable again.
- Catch-up order. Decide before the incident which systems catch up first, for instance payments and fraud checks before internal reporting.
In our reference architecture, Gateway rate limits pace the applications that replay writes, and its produce policies enforce acks=all and idempotence, so a misconfigured producer can't lose acknowledged writes or duplicate them on retry.
The Architecture to Survive a Region Loss
The blueprint has one design per failure:
- Losing Kafka: Gateway switches to the DR cluster.
- Losing the Gateway deployment: a standby Gateway on the same Kafka takes over.
- Losing a whole region: the design below, across two cloud regions and the bank's own data center.
- Same address everywhere.
gateway.mycompany.comresolves to Gateway A in Region A, to the standby Gateway in Region B, and to the F5 BIG-IP pair in the data center. Applications never change their bootstrap address. - A standby that starts on demand. The standby Gateway runs no instance until a region is lost. Then the runbook cuts off the old region, checks that Gateway's configuration (service accounts, interceptors) is on the secondary, and starts the standby and the payments-app copy. Keeping it stopped prevents writes to both clusters at once and avoids paying for cross-region traffic. If that cold start is too slow for your recovery target, keep the standby running.
The blueprint generates that runbook from the design itself: one script with read-only checks, confirmed steps for each failover, fence and failback, and a drill for pre-production.

Choosing Your Trade-offs
Making a bank's Kafka platform resilient is hard: no single team owns the whole path of a payment. The fault can sit in the data, the code, a site or a counterparty, replication always leaves a gap, and recovery depends on ledgers and partners that Kafka doesn't control. No tool makes an outage disappear.
So choose your trade-offs with the business, before the incident:
| Choice | You gain | You pay |
|---|---|---|
| Stretch cluster | No data loss when a site dies | Latency on every write |
| Two clusters | Fast writes | Seconds of payments at risk |
| Warm standby | Fast recovery | Money every day |
| Cold standby | Lower cost | Minutes during the outage |
- How much data you can lose: your RPO.
- How long you can be down: your RTO (recovery time).
- Who decides on the incident call.
To work through these trade-offs on your platform, book 45 minutes with our engineers: we'll map one of your flows in the blueprint, and you leave with its design and open issues.
