How to Manage Kafka Topics and Schemas Easily

The payments team needs a new topic, payments.refunds. A platform engineer runs kafka-topics --create with six partitions and seven days of retention. A week later the first producer deploys, and its serializer registers a schema under payments.refunds-value on its own.

Nothing in that sequence is wrong, and it's how a lot of teams create topics. But the topic and its schema came from two different tools, at two different times, under two different sets of permissions. The only thing connecting them is that one name starts with the other.

That split is why managing topics and schemas feels harder than it should. The easy way isn't a nicer tool for each half. It's treating a topic and its schemas as one resource: declared together, owned by the same team, and checked before either one reaches the cluster.

Kafka and the schema registry each do their job well

A topic lives on the brokers. You create it with kafka-topics or the AdminClient, and you give it a partition count, a replication factor and topic configs like retention.ms and cleanup.policy.

A schema registry is a separate service with its own REST API. It stores versioned schemas under a subject, and it checks each new version against the subject's compatibility mode, so a producer can't publish a change that breaks the consumers reading it.

Here's what one topic and its value schema touch on each side:

Change	On the Kafka side	On the registry side
Create it	kafka-topics --create	POST /subjects/payments.refunds-value/versions
Set its rules	retention.ms, min.insync.replicas	PUT /config/payments.refunds-value for compatibility
Control who changes it	Kafka ACLs on the topic	The registry's own auth, if it has any
Delete it	kafka-topics --delete	DELETE /subjects/payments.refunds-value, as a separate call

Both halves are sound. Teams with good automation script them side by side, or keep topics in Terraform and schemas in a CI job, and for a handful of teams that holds up fine.

The broker doesn't know a topic has a schema

Kafka doesn't store any link between a topic and a schema. To the broker, a record is bytes. The link lives in the client's serializer, which by default derives the subject from the topic name: with TopicNameStrategy, payments.refunds maps to payments.refunds-value.

Because a name is the only link, the two sides drift apart in predictable ways:

• Topics without schemas. A topic created by hand has no schema until a producer registers one, and on most clusters nothing stops the first producer from sending plain JSON instead.
• Schemas without topics. Deleting a topic with kafka-topics --delete doesn't touch the registry. Its subjects stay behind, and a new topic with the same name picks up the old schema history.
• Two access models. Kafka ACLs decide who can write to the topic, and the registry decides separately who can change what its records look like. That second half is often wide open, as we covered in Kafka Schema Registry: Open by Default.
• Other naming strategies. With RecordNameStrategy, the subject is named after the record type, not the topic, and the name match stops working altogether.

"Our producers auto-register schemas"

🚫 "Our producers auto-register schemas, so schemas take care of themselves."

Auto-registration is convenient, and in development it's often the right default. With auto.register.schemas=true, the default in Confluent's serializers, the first producer to write a new shape registers it, and the registry checks it against the subject's compatibility mode.

But that hands the schema decision to whichever deploy runs first. The compatibility mode is the registry's global default unless someone set one on the subject, and nobody reviewed the change, because nobody proposed it. It happened as a side effect of a deploy.

In the conversations we have with platform teams, creating a topic often means a ticket to a central team. A schema registered by a producer's first deploy skips that step entirely. That's the wrong way round: retention is easy to change later, and a schema that consumers already depend on isn't.

Manage the topic and its schemas as one change

The simpler model is to stop treating them as two resources. A topic and its key and value subjects are one unit, and every change to that unit goes through one path:

• Declared together. The topic's config and its schema files sit in the same folder of the same repository, so a pull request that adds a field shows the schema diff next to the topic it belongs to.
• Owned together. The team that owns the payments. topics owns the payments. subjects too, and nobody else can create or change either one.
• Checked together. The topic config goes through the platform's rules for replication, retention and naming, and the new schema goes through the registry's compatibility check, in the same dry run.
• Blocked together. A pull request that fails either check doesn't merge, so the cluster never receives half of a change.

Other tools cover parts of this. Strimzi reconciles topics declared as KafkaTopic resources, and Terraform providers can manage topics and, in some cases, schemas. What they don't record is who owns a prefix, so ownership stays a convention in the repository layout.

Conduktor Console manages both as owned resources

We built this into Conduktor Console. Topics and subjects are both resources you apply with the Conduktor CLI, so the two can sit in the same file:

---
apiVersion: kafka/v2
kind: Topic
metadata:
  cluster: prod
  name: payments.refunds
spec:
  replicationFactor: 3
  partitions: 6
  configs:
    retention.ms: '604800000'
    min.insync.replicas: '2'
---
apiVersion: kafka/v2
kind: Subject
metadata:
  cluster: prod
  name: payments.refunds-value
spec:
  schemaFile: schemas/refunds.avsc
  format: AVRO
  compatibility: FORWARD_TRANSITIVE

The Topic goes to the brokers and the Subject to the registry connected to the same cluster. schemaFile points at the Avro file in the repository, so the schema is reviewed like code, and compatibility is set on the subject instead of inherited from the global default.

Ownership comes from federated ownership. A platform team declares an application instance that owns the payments. prefix for topics, consumer groups and subjects alike. Anything outside the prefix is rejected before other checks run, and no two instances can claim overlapping prefixes on a cluster.

The platform's rules are resource policies, written as CEL conditions:

• Topic policies can require a replication factor of 3, keep retention.ms inside a range, or enforce a naming pattern.
• Subject policies can allow only Avro or Protobuf, or require a specific compatibility mode.
• conduktor apply --dry-run checks the topic against Kafka with validateOnly and the subject against the registry's compatibility API, so a CI job that runs it fails the pull request before anything reaches the cluster.

In the UI, each topic's Schema tab shows its key and value subjects, and the Schema Registry view compares versions and checks compatibility before an update.

What it doesn't do. The Schema tab finds subjects through TopicNameStrategy, so subjects named after a record type don't show up against the topic. Admin API keys skip resource policies by design, so keep them out of application pipelines. And a policy can't compare a resource with its previous version, so set a fixed compatibility mode per environment instead of a rule like "only stricter".

Where to start

You don't have to move every topic at once. An order that works:

1. Pick one naming strategy and write it down. TopicNameStrategy keeps the link between a topic and its subjects visible, and it's what most tooling expects.
2. Give every prefix an owner. The team that owns payments. topics should own payments. subjects too.
3. Put schemas next to the topics. Review a schema change in the same pull request as the topic it describes.
4. Set compatibility per subject. Don't rely on the registry-wide default for production subjects.
5. Turn off auto-registration in production. Keep it for development, and let production schemas arrive through the pipeline.
6. Then clean up the drift. List topics with no schema and subjects with no topic, and decide for each one whether it should still exist. Console's governance Insights shows how many topics have a registered schema, which is a good place to begin.

Topics and schemas are hard to manage together because Kafka and the schema registry don't know about each other, and the only link between them is a subject name. Managing them easily means giving each topic and its schemas one owner, one place in the repository and one check before anything is applied, so a topic and the shape of its data change in the same reviewed step.

See how federated ownership works in Console →

---

Related: Kafka Schema Registry: Open by Default → · How to Manage Kafka ACLs and Security at Scale → · Terraform Your Whole Kafka Platform →