MSK to Confluent Cloud Migration: A Step-by-Step Guide

The hard part of Kafka migration isn't replication. It's the cutover: switching producers and consumers at the right moment, with the right offsets, without duplicating or losing data.

I've helped teams through a dozen migrations. The replication always works. The coordinated cutover is where things go wrong.

Our first migration attempt lost 12 hours of data because consumer offsets didn't sync. Second attempt, with proper checkpoint configuration, was seamless.

SRE at a payments company

Pick Your Tool

Tool	Cost	Offset Sync	Lock-in
MirrorMaker 2	Free	Requires config	None
Cluster Linking	Confluent license	Built-in	Confluent-only

This guide uses MirrorMaker 2 because it works for any Kafka-to-Kafka migration. You can monitor and manage MirrorMaker connectors through a Kafka Connect UI.

Basic MM2 Configuration

# mm2.properties
clusters = msk, confluent

msk.bootstrap.servers = b-1.msk-cluster.amazonaws.com:9096
msk.security.protocol = SASL_SSL
msk.sasl.mechanism = AWS_MSK_IAM
msk.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required;

confluent.bootstrap.servers = pkc-xxxxx.confluent.cloud:9092
confluent.security.protocol = SASL_SSL
confluent.sasl.mechanism = PLAIN
confluent.sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="KEY" password="SECRET";

# Replication flow
msk->confluent.enabled = true
msk->confluent.topics = .*

# Keep same topic names (Kafka 3.0+)
replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy

Enable Consumer Offset Sync

Without offset sync, consumers restart from beginning or end after cutover.

msk->confluent.sync.group.offsets.enabled = true
msk->confluent.emit.checkpoints.enabled = true
msk->confluent.sync.group.offsets.interval.seconds = 10
msk->confluent.groups = .*
offset-syncs.topic.location = target

Tradeoff: Offset sync only works for consumer groups inactive on the target cluster.

Cutover Sequence

This is where migrations fail. The order matters:

1. Stop producers to MSK
2. Wait for replication lag → 0
3. Stop consumers on MSK (let them commit final offsets)
4. Wait 30-60 seconds for checkpoint connector to sync
5. Start consumers on Confluent Cloud
6. Start producers to Confluent Cloud

# Verify consumers are at expected positions
kafka-consumer-groups.sh \
  --bootstrap-server pkc-xxxxx.confluent.cloud:9092 \
  --describe --group orders-processor
# LAG should be near 0 if offsets synced correctly

Common Migration Failures

Consumer resumes from wrong offset:

Offset sync didn't complete, or consumer was active on both clusters. Manually reset using values from checkpoint topic:

kafka-consumer-groups.sh --bootstrap-server pkc-xxxxx.confluent.cloud:9092 \
  --group orders-processor --topic orders \
  --reset-offsets --to-offset <translated-offset> --execute

IAM authentication errors:

Ensure the IAM auth JAR is in classpath:

cp aws-msk-iam-auth-2.0.3-all.jar $KAFKA_HOME/libs/

Rollback Plan

Set up reverse replication before migration:

1. Stop producers to Confluent Cloud
2. Configure MM2 to replicate Confluent → MSK
3. Wait for new data to sync back
4. Switch back to MSK

Don't scramble to configure rollback during an incident.

Migration is operationally complex because you're coordinating multiple teams, multiple services, and multiple failure modes. The replication itself is the easy part.

Book a demo to see how Conduktor Console provides unified visibility across MSK, Confluent Cloud, and self-hosted clusters.