A Kafka migration plan usually starts with a replication tool and a cutover date. The tool gets a pattern that matches every topic, and the platform team then asks each application team to sign off on moving its share. Some of those teams have never heard of the topics on their list.
"Why are you telling us to migrate 20 topics that you've never accessed?"
Technology engineer at an asset manager, describing the pushback from application teams
A migration should start by deciding which topics move, each with a team that owns it and evidence that someone reads it. Replication starts after that decision, and it copies only those topics.
Replication Tools Copy the List You Give Them
Every replication tool takes a list or a pattern of topics and copies what matches. MirrorMaker 2, the open-source option, reads it from topics, a comma-separated list of regular expressions whose default is .*, so a flow you configure without it mirrors every topic except the internal ones that topics.exclude filters out. Managed replication services work the same way: you give them a filter, and they apply it.
None of them decides whether a topic is worth moving. That decision costs little before replication starts and a lot after, because each topic on the list adds work at every later step:
| Each topic you move adds | Where the work lands |
|---|---|
| Partitions to copy and replication lag to watch | The platform team, for the whole parallel run |
| Schema subjects to carry over with their IDs | Whoever owns the schema registry move |
| Service accounts and ACLs to recreate | Security and the platform team |
| Consumer offsets to translate or re-seed | Every team with a consumer on it |
| Clients to repoint at the new cluster | Every team with a producer or consumer on it |
| A sign-off before cutover | The team that owns it, if there is one |
Kafka Migration Tools: What Each One Actually Moves covers how each tool handles offsets, schemas and identities once the list exists.
Kafka Doesn't Record Who Reads a Topic
Shrinking the list means proving, topic by topic, that nobody needs it. Kafka makes that harder than it sounds, because the broker keeps no history of who read or wrote a topic. It keeps three partial records:
- Consumer groups. A consumer group stores its committed offsets on the broker, so listing groups shows who is subscribed. Readers that assign partitions manually and don't commit never show up.
- ACLs. They say which principal may read or write a topic. They don't say whether that principal ever connected.
- Broker metrics. Brokers expose bytes in and out per topic as live metrics. They show traffic only while something scrapes and keeps them, and they don't name the client.
Producers leave the least trace: a producer that connects, writes and disconnects leaves its records and nothing that names the application that sent them. How to Find the Producers and Consumers of a Kafka Topic walks through every source of evidence and what each one misses.
The consumer-group record also expires. Once a group has no active members, the broker keeps its committed offsets for offsets.retention.minutes, which defaults to seven days, then deletes them. Take a reconciliation job that runs once a month:
An inventory taken in those three weeks lists the topic as unread. Retire it on that evidence, and the job finds no data on the new cluster the next time it runs.
🚫 "No consumer group on a topic means nobody reads it."
A missing group proves only that nobody committed offsets in the last week. Monthly jobs, consumers with manual assignment, and readers that keep their offsets outside Kafka all look the same as no reader at all. The broker evidence can rule a topic in, but it can't rule one out: only the team that owns the topic can do that.
Sort Every Topic Into Move, Retire or Ask
The broker evidence is enough to sort topics into four groups, and only one of them gets replicated without a conversation:
Each group needs a different decision:
- Empty. The topic holds no records, often because it was created for a test or a project that never shipped. There's nothing to replicate. If a team turns out to need it, it recreates the topic on the new cluster from its configuration.
- Stale. No recent writes and no active consumer group. A topic filled by a monthly batch, the dead letter topic that fills only on errors, and the data kept for an audit all land in this group. Ask the owning team, then either move it or leave it behind.
- Live, no owner. Records arrive or a group reads them, but no team claims the topic. It can't move until one does, because nobody else can repoint its clients.
- Live, owned. The move list. These are the topics the replication filter names.
The sort also documents the cluster for the teams that stay on it:
"As we migrate these applications to our new clusters, I'm taking that opportunity to kind of do some documentation and grouping on our topics…"
System architect at a manufacturer
Give Every Topic on the List an Owner Before Replication Starts
Ownership is what turns the list into a plan. The team that owns a topic signs off on moving it, tells the platform team who reads it, and agrees on a cutover date with those readers. When no team owns a topic, the platform team ends up chasing each reader itself.
"Given the topic, I have to hunt down… who all is consuming from this topic."
Platform engineering manager at a software company
Two steps keep that hunt short:
- Record ownership where the platform can read it. A label, a naming prefix that maps to a team, or a catalog entry, so it outlives the migration project.
- Check the owner's list against the ACLs. The owner knows who asked for access. The ACLs show every principal that was granted read or write, including the ones granted before the current team took over.
Once a topic has an owner and a list of readers, the order of the move follows: readers move to the new cluster first, while replication keeps their data current, and the producer moves last.
Write the Replication Filter Last
With the list settled, the replication configuration names it explicitly. In MirrorMaker 2, that means a pattern per team prefix instead of .*, exclusions for what stays behind, and a replication policy that keeps topic names unchanged so readers can move before producers:
source->target.enabled = true
source->target.topics = payments\..*, orders\..*, shipping\..*
source->target.topics.exclude = .*[\-\.]internal, .*\.replica, __.*, .*\.tmp, .*-test
replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy Setting topics.exclude replaces its default, so the first three patterns put back the internal topics MirrorMaker 2 skips on its own. A prefix per owning team keeps the filter short and makes each line answer to one team. Topics that stay behind remain on the old cluster, readable, until it shuts down. A reader nobody found can still come forward until then.
Building the List in Conduktor Console
Conduktor Console covers the inventory and the ownership steps. Its Insights cost control view lists empty topics, stale topics (no writes for seven days or more and no active consumer group) and tiny ones, with idle days and partition count for each, and each topic's page shows its consumer groups and produce activity. Treat the stale list as the topics to ask about, for the reasons above.
Stream lineage maps, for every service account, the topics its ACLs or role bindings let it read and write, so you see who to ask about each one. With federated ownership, applications own their topics and other teams request access to them, so the topics on the move list arrive with a team attached and the readers it approved.
A migration moves whatever list it's given, and Kafka can't tell you on its own which topics still matter. Build the list first: owned, read, and agreed with the teams that read it. The new cluster then starts with only the topics teams use and a named owner for each, which is also the inventory the next migration will need.
Related: Kafka Migration Tools: What Each One Actually Moves → · How to Find the Producers and Consumers of a Kafka Topic → · How Do I Reduce Our Kafka Costs? →
