Kafka Disaster Recovery: The Complete Strategy Beyond Replication

In the previous post on why Kafka replication isn't the hard part, we made the case that replication is the solved problem. Most teams already have MirrorMaker, Cluster Linking, or something similar running between clusters. The hard part is everything else: switching clients under pressure, maintaining security and governance through failover, and executing a runbook that half the team has never seen.

A complete Kafka disaster recovery strategy covers six technical areas beyond data replication, organized around three operational phases. Most organizations have invested heavily in one or two. The ones that recover in minutes instead of hours have addressed all six, and they've centralized the solutions rather than stitching together per-service workarounds.

This post is the narrative companion to our Disaster Recovery Readiness Checklist. The reasoning behind what's on the list, the common gaps, and the practical tips that come from seeing these plans succeed and fail.

Three phases of Kafka disaster recovery readiness

Disaster recovery isn't a switch you flip. It's a discipline with three distinct phases, each requiring different work. Getting replication right covers roughly a third of it.

Phase 1: Prepare

The work you do before anything breaks determines whether recovery is even possible.

• Define RTO/RPO per domain and workload. Not every topic has the same criticality. Payments, authentication, and critical ETL have different recovery requirements than analytics dashboards. Classify workloads into tiers and set targets for each.
• Choose per service: standby cluster (active/passive) or both clusters serving traffic (active/active). This is a per-workload decision, not a global one. Some services justify the cost and complexity of active/active. Most don't.
• Set up cross-cluster data replication. Whether it's MirrorMaker, Cluster Linking, or something else, replication is the foundation that everything else builds on.
• Map your dependencies. Applications, topics, schemas, ACLs, certs, quotas: you need a clear picture of what moves together and what can wait. This dependency map is what turns a chaotic scramble into a sequenced plan.
• Identify your Wave 1 applications. Define which services must recover first (payments, authentication, critical ETL) and build your disaster recovery plan around recovering these before the long tail.

Tip: If Wave 1 recovers in minutes, you've bought yourself time for everything else. Keep the Wave 1 list short and explicit. If it has more than a dozen services on it, you haven't prioritized, you've just renamed the full list.

Phase 2: Validate

A disaster recovery plan that hasn't been tested isn't a plan, it's a hypothesis. This is where most organizations fall short.

• Run chaos tests. Simulate broker loss, region loss, auth failures. Start in staging, graduate to production game days. Quarterly at minimum for mission-critical workloads.
• Test your monitoring and decision chain, not just your infrastructure. Can you detect a failure without relying on the infrastructure that just failed? Do the right people get paged? Can the person on call at 3 AM actually authorize a failover, or do they need an approval chain that takes 30 minutes?
• Pre-stage and verify disaster recovery configurations in both regions. Topic configs, ACLs, schemas, certificates, quotas: all of it should be in place and verified before you need it. Configuration drift between primary and secondary is one of the most common silent disaster recovery failures.

Tip: Measure your actual failover time during drills. If your RTO target is 15 minutes but your drill takes 90, you don't have a disagreement, you have a gap. Most auditors will ask for this number.

Phase 3: Execute

When disaster strikes, the quality of your preparation and validation determines whether recovery takes minutes or hours. Execution should be a sequence, not an improvisation.

• Switch clients through a stable endpoint, not per-app config rewires. If failover requires touching dozens of services individually, your execution time scales with your service count, not with the quality of your tooling.
• Enforce consistent security and governance during cutover. The moment you're most likely to make a security mistake is the moment you're moving fastest under the most stress.
• Communicate. Failover is an organizational event, not just a technical one. Stakeholders, downstream consumers, and compliance teams need to know what happened and what changed.

Tip: Never rely on humans for the mechanical parts of execution. Reserve human judgment for the decision to fail over, then let tooling handle the act of failing over. Systems don't stress at 3 AM, forget steps, or need approval chains.

Six areas to verify in your Kafka disaster recovery plan

The three phases above are the operational framework. Below are the six technical areas that underpin them, the specific things your disaster recovery design review should verify.

1. Security and identity parity

What it covers: TLS/cert strategy including custom CAs, auth parity across regions (mTLS/SASL/OAuth), ACL/RBAC parity and least-privilege maintenance, audit log availability and retention.

What goes wrong: Teams replicate data but not the security posture around it. Auth credentials are often cluster-specific (Confluent Cloud Kafka API keys, for instance, are tied to individual clusters). ACLs may exist on the primary but were never provisioned on the disaster recovery cluster. During failover, apps authenticate but can't authorize or get broader access than intended, creating compliance exposure at the worst possible moment.

Tip: If clients authenticate directly to Kafka, every client needs credentials for both clusters. If a proxy layer like Conduktor Gateway decouples client identity from cluster credentials, only the proxy needs dual-cluster access. This architectural simplification pays off well beyond disaster recovery.

2. Topic and schema configuration parity

What it covers: Topic config parity (retention, compaction, quotas), Schema Registry parity and compatibility rules.

What goes wrong: Replication tools move data, not configuration. Topic-level settings drift between clusters over time. Schema Registry is often a separate system entirely where schemas may not be replicated, or compatibility rules may differ. After failover, consumers fail on schema mismatches, topics compact unexpectedly, and quotas throttle catch-up traffic right when you need maximum throughput.

3. Data protection and compliance continuity

What it covers: Encryption and masking parity (field-level, payload-level), PII and regulated data handling consistency across regions.

What goes wrong: Encryption and masking policies are often applied at the application or infrastructure layer, not within Kafka. When clients fail over to a different cluster or region, those policies don't follow, especially if they're enforced by infrastructure co-located with the failed primary. The result is PII exposure during failover, at a moment already attracting regulatory and executive scrutiny.

4. Observability and operational readiness

What it covers: Monitoring for replication lag, broker health, and client error rates. Capacity planning for post-failover catch-up traffic. Producer timeout and backpressure configuration.

What goes wrong: Monitoring is often configured per-cluster. If your dashboards, alerts, and on-call routing depend on the same infrastructure that failed, you're flying blind during failover. Separately, teams rarely plan for the burst of catch-up traffic that hits the secondary, leading to cascading failures on what was supposed to be the rescue.

RPO has two sides. Cross-cluster replication lag tells you how far behind the secondary is. But producer configuration determines how much data you lose on the primary side during an outage:

• Producers buffer unsent records in memory. When brokers go down, that buffer fills up, new sends block, and eventually both blocked and buffered records fail with timeout exceptions. If your application doesn't handle these errors, messages are lost.
• The default delivery timeout is two minutes. If your detection and decision window is longer than that (and for most organizations, it is), producers will silently discard data before anyone has decided to fail over.
• Backpressure design matters. Producer timeouts, buffer sizes, and how upstream applications handle blocked sends all need to be tuned against your actual expected failover timeline.

Tip: Plan secondary cluster capacity for at least 1.5x normal load to absorb the reconnection burst. And review your producer timeout settings: if delivery.timeout.ms is shorter than your expected detection-plus-decision window, your actual RPO is worse than your replication lag suggests.

5. Client switching and traffic routing

What it covers: How clients get routed from primary to secondary during failover, whether that requires per-app config changes and restarts or can happen from a single control point.

What goes wrong: There is no mechanism built into Kafka to redirect running clients during failover. Every workaround, whether centralized Kubernetes operators, DNS-based service discovery, or custom wrapper code, either requires centralized control of all clients (which most organizations don't have at scale) or bespoke code in every application. Most teams invest in detection, but the real bottleneck is usually execution, because it requires touching many things simultaneously with no single point of control.

What good looks like: a stable endpoint that clients connect to once, with cluster switching handled behind it. The failover becomes a single operational decision, not a per-service coordination exercise across teams and repos. This is the pattern that Conduktor Gateway implements: applications connect to Gateway, and the underlying cluster can be switched without touching any client configuration.

Tip: Audit your Kafka client versions. Kafka 3.8 introduced client-side rebootstrap (KIP-899), an opt-in feature that lets clients fall back to bootstrap servers when all discovered brokers fail. Older clients can't, they get stuck and require restarts. Factor this into your RTO estimates.

6. Testing and continuous validation

What it covers: Regular chaos testing (quarterly at minimum for critical workloads), game days simulating regional loss, and validating that runbooks are current and executable by whoever is on call.

What goes wrong: Disaster recovery testing is the most commonly skipped step. It's disruptive, expensive, and politically difficult, so it happens rarely or not at all. Untested disaster recovery is functionally equivalent to no disaster recovery: you've invested in infrastructure without validating it works under pressure. The ability to inject failures at the proxy layer (simulating broker errors, auth failures, latency) without touching production Kafka lowers the barrier significantly.

Tip: Treat your runbook like code: version it, review it, test it on a schedule. If the person who wrote it left six months ago and nobody has reviewed it since, it's not a runbook, it's a liability.

The failover runbook: a simple sequence

If preparation and validation are done well, actual failover should be a calm, sequenced procedure, not an improvisation. Here's what that looks like.

The goal: no heroics. This should be simple enough that the person on call at 3 AM can execute it without the person who designed it.

Tip: Define your Wave 1 applications explicitly and validate them first in every game day. If your drill only proves the infrastructure can switch but doesn't verify critical applications actually recovered, you've tested plumbing, not disaster recovery.

From checklist to action

Most teams have solved replication plus one or two of the areas above. The organizations that recover quickly have addressed all six, and they've found ways to centralize the solutions rather than solving each independently across every service and team.

Each area solved in isolation means a different tool, a different team, a different maintenance burden. A platform layer that addresses multiple areas simultaneously - security decoupling, centralized switching, policy continuity, and chaos testing - compresses the investment and simplifies the operational model.

Download the Disaster Recovery Readiness Checklist and bring it to your next design review or disaster recovery audit.

Book a Disaster Recovery Workshop. 45 minutes to review your Kafka estate and build a first-pass wave plan.

This is part of a series on Kafka Disaster Recovery.

Previously: Kafka DR: Why Data Replication Isn't the Hard Part

Read next: How Gateway Reduces Kafka Disaster Recovery from Hours to Minutes