Every Kafka disaster-recovery review ends on the same question: how do you do failover with Kafka, and why isn't it automatic? The answer is that failover is two different jobs, and most designs automate the wrong one.
The first job is deciding to abandon the primary cluster, which depends on replication state you can't fully see. The second is moving every client to the secondary, which is mechanical but which most teams still do by hand, one application at a time.
| Deciding to fail over | Moving the clients | |
|---|---|---|
| Depends on | Replication lag, state of the primary | Nothing: it's the same steps every time |
| Usually done by | A health check | A war room, one application at a time |
| Should be | A person | One call |
Two Clusters Means Data Loss and Duplicates
An active-passive Kafka setup needs something copying the primary into the secondary. Kafka brokers don't replicate to another cluster on their own, so you run MirrorMaker 2 (shipped with Kafka), Confluent Replicator, Confluent Cluster Linking or Redpanda Shadow. All of them replicate asynchronously, which means whatever the primary accepted in the seconds before it died hasn't reached the secondary yet. A recovery point objective (RPO) of zero isn't achievable on a two-cluster design, whichever tool you pick.
The second problem is offsets. A record at offset 4,000,000 on the primary might be offset 12 on a secondary that was created last month, and the two categories of replication tool deal with that gap differently.
| Tool | How it replicates | Offsets on the secondary |
|---|---|---|
| MirrorMaker 2, Confluent Replicator | Kafka Connect: a consumer reads the primary, a producer writes the secondary | Different. Consumer positions are translated: MM2 from offset-sync checkpoints, Replicator by timestamp |
| Confluent Cluster Linking, Redpanda Shadow | Broker-level: the secondary mirrors the primary's log without a producer in between | Preserved |
The Decision to Fail Over Stays Manual
Active-passive Kafka works on one rule: exactly one cluster is live at a time. Two live clusters is worse than a dead one, because producers write to both, consumers read from both, and the replication link keeps copying the primary's new writes into a secondary that's also taking writes of its own, so the two diverge.
An automatic trigger creates exactly that. A health check sees the primary as down and flips traffic, but "down" is usually "degraded", and some clients still hold connections to the old primary and keep producing there. A consumer that reads the same order from both clusters processes it twice. Data loss costs you records; double processing costs you money, and it's much harder to unwind.
The trigger also can't see replication lag. If MirrorMaker was two minutes behind when the primary went down, switching now loses two minutes of data, and whether that's acceptable depends on whether the primary might come back in ten minutes. No threshold can make that call for you.
The replication vendors say the same thing in their own docs. Confluent's Cluster Linking guide states that failing over your applications is your responsibility and that lagged data may not have reached the destination. Redpanda's Shadowing docs call failover irreversible and note that automatic fallback to the original cluster isn't supported. Both ship a promote command and leave the decision to run it with you.
So the runbook has a fixed order, and only the last step is mechanical:
- Try to bring the primary back.
- Decide you can't.
- Make sure the primary can't accept writes.
- Stop the replication link.
- Move the clients.
Stopping the replication link looks different per tool, and for Cluster Linking and Shadow it's the same command that makes the secondary writable:
| Tool | How you stop replication | Consumer offsets after |
|---|---|---|
| MirrorMaker 2 | Stop or pause the source connector | Translated from checkpoints |
| Confluent Replicator | Stop the connector | Translated by timestamp |
| Confluent Cluster Linking | confluent kafka mirror failover promotes mirror topics; irreversible | Preserved |
| Redpanda Shadow | rpk shadow failover --all promotes shadow topics; irreversible | Preserved |
How You Do Failover With Kafka
Recovery time (RTO) breaks down into detecting the failure, deciding to switch, and executing the switch. Detection is your alerting and the decision is a person confirming the primary is gone, so execution is where the hours actually go. A team running 200 Kafka applications has 200 sets of bootstrap servers, credentials, owners and restart procedures to work through.
Kafka clients hold long-lived connections and keep retrying against the brokers they already know. A client retrying against a dead cluster won't re-resolve DNS on any schedule you control, and Java's DNS cache is held forever when a security manager is installed, so a DNS flip leaves every team checking whether their service actually noticed. We covered the DNS and load-balancer problems in detail in How Conduktor Gateway reduces Kafka DR from hours to minutes.
Credentials are a separate step. The secondary has to authenticate and authorize every client before the switch, and on Confluent Cloud, where API keys are scoped to a single cluster, a client that fails over also needs a new secret.
The fix is to give every client one address that never changes and move the target behind it. With Conduktor Gateway, clients connect to Gateway and authenticate with Gateway, and which physical cluster sits behind it is Gateway configuration rather than client configuration. The switch itself is one HTTP call per Gateway instance (Gateway failover docs):
curl --request POST 'http://gateway:8888/gateway/v2/cluster-switching' \
--header 'Content-Type: application/json' \
--data-raw '{ "fromPhysicalCluster": "main", "toPhysicalCluster": "failover" }'Gateway closes the client connections, and the clients reconnect to the same bootstrap address, refresh their metadata, and find themselves talking to the secondary. Consumers rejoin their group and resume from the replicated offsets, producers send what they'd been retrying, and no application was restarted or reconfigured along the way.
The switch has two constraints:
- The call goes to every Gateway instance. A script that iterates the instances belongs in the runbook, so the switch is still one step for the operator.
- Clients need retries long enough to cover the decision. That's the one part no proxy can handle for you.
What the Clients Still Have to Handle
Gateway makes the switch, but the applications still have to be ready for it.
- Producers drop records after two minutes. The Java producer keeps unsent records for
delivery.timeout.ms, 120,000 ms by default, and then fails them, so if the decision takes longer than that, those records are failed back to the application before the switch happens. Either raise the timeout to cover your realistic decision time or make the application fail and stop accepting work. What it can't do is keep returning success while batches expire. - Consumers can see records twice. Translated offsets make re-reads likely, and even offset-preserving replication can leave a committed position ahead of the data that actually arrived. Idempotent processing is a requirement.
- Credentials must already exist on the secondary. With Gateway, clients authenticate against Gateway and Gateway holds the credentials for each physical cluster, so the switch doesn't hand anyone a new secret. Without it, this is a pre-provisioning job for every application.
- Failback is a second failover. Once the old primary is repaired, it's the one that's behind, so replication runs the other way and the same runbook applies in reverse.
🚫 "We'll automate failover off the broker health check and get RTO to zero."
Automate the Switch, Not the Decision
Keep the decision to fail over manual. Someone has to check replication lag and confirm the primary is dead, and no health check can do that for you.
Automate everything after it. Put a proxy in front of Kafka so the switch is one call, clients reconnect on their own, and nobody has to touch 200 applications during an outage.
Related: How Conduktor Gateway Reduces Kafka DR from Hours to Minutes → · Kafka DR: Why Replication Isn't the Hard Part → · Disaster Recovery Strategies for Kafka Clusters →