# How to Manage Kafka Topics and Schemas Easily

The payments team needs a new topic, `payments.refunds`. A platform engineer runs `kafka-topics --create` with six partitions and seven days of retention. A week later the first producer deploys, and its serializer registers a schema under `payments.refunds-value` on its own.

Nothing in that sequence is wrong, and it's how a lot of teams create topics. But the topic and its schema came from two different tools, at two different times, under two different sets of permissions. The only thing connecting them is that one name starts with the other.

That split is why managing topics and schemas feels harder than it should. The easy way isn't a nicer tool for each half. It's treating a topic and its schemas as one resource: declared together, owned by the same team, and checked before either one reaches the cluster.

## Kafka and the schema registry each do their job well

A [topic](https://www.conduktor.io/glossary/kafka-topics-explained) lives on the brokers. You create it with `kafka-topics` or the AdminClient, and you give it a partition count, a replication factor and [topic configs](https://kafka.apache.org/documentation/#topicconfigs) like `retention.ms` and `cleanup.policy`.

A [schema registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management) is a separate service with its own REST API. It stores versioned schemas under a subject, and it checks each new version against the subject's compatibility mode, so a producer can't publish a change that breaks the consumers reading it.

Here's what one topic and its value schema touch on each side:

| Change | On the Kafka side | On the registry side |
|---|---|---|
| Create it | `kafka-topics --create` | `POST /subjects/payments.refunds-value/versions` |
| Set its rules | `retention.ms`, `min.insync.replicas` | `PUT /config/payments.refunds-value` for compatibility |
| Control who changes it | Kafka ACLs on the topic | The registry's own auth, if it has any |
| Delete it | `kafka-topics --delete` | `DELETE /subjects/payments.refunds-value`, as a separate call |

Both halves are sound. Teams with good automation script them side by side, or keep topics in Terraform and schemas in a CI job, and for a handful of teams that holds up fine.

## The broker doesn't know a topic has a schema

Kafka doesn't store any link between a topic and a schema. To the broker, a record is bytes. The link lives in the client's serializer, which by default derives the subject from the topic name: with `TopicNameStrategy`, `payments.refunds` maps to `payments.refunds-value`.

Two systems, two clients, two permission models. The subject name is the only thing that ties them together.

Because a name is the only link, the two sides drift apart in predictable ways:

- **Topics without schemas.** A topic created by hand has no schema until a producer registers one, and on most clusters nothing stops the first producer from sending plain JSON instead.
- **Schemas without topics.** Deleting a topic with `kafka-topics --delete` doesn't touch the registry. Its subjects stay behind, and a new topic with the same name picks up the old schema history.
- **Two access models.** Kafka ACLs decide who can write to the topic, and the registry decides separately who can change what its records look like. That second half is often wide open, as we covered in [Kafka Schema Registry: Open by Default](https://www.conduktor.io/blog/kafka-schema-registry-open-by-default).
- **Other naming strategies.** With `RecordNameStrategy`, the subject is named after the record type, not the topic, and the name match stops working altogether.

## "Our producers auto-register schemas"

> 🚫 *"Our producers auto-register schemas, so schemas take care of themselves."*

Auto-registration is convenient, and in development it's often the right default. With `auto.register.schemas=true`, the default in Confluent's serializers, the first producer to write a new shape registers it, and the registry checks it against the subject's [compatibility mode](https://www.conduktor.io/glossary/schema-evolution-best-practices).

But that hands the schema decision to whichever deploy runs first. The compatibility mode is the registry's global default unless someone set one on the subject, and nobody reviewed the change, because nobody proposed it. It happened as a side effect of a deploy.

In the conversations we have with platform teams, creating a topic often means a ticket to a central team. A schema registered by a producer's first deploy skips that step entirely. That's the wrong way round: retention is easy to change later, and a schema that consumers already depend on isn't.

## Manage the topic and its schemas as one change

The simpler model is to stop treating them as two resources. A topic and its key and value subjects are one unit, and every change to that unit goes through one path:

- **Declared together.** The topic's config and its schema files sit in the same folder of the same repository, so a pull request that adds a field shows the schema diff next to the topic it belongs to.
- **Owned together.** The team that owns the `payments.` topics owns the `payments.` subjects too, and nobody else can create or change either one.
- **Checked together.** The topic config goes through the platform's rules for replication, retention and naming, and the new schema goes through the registry's compatibility check, in the same dry run.
- **Blocked together.** A pull request that fails either check doesn't merge, so the cluster never receives half of a change.

One folder, one dry run, two destinations. A failed check stops both halves.

Other tools cover parts of this. [Strimzi](https://strimzi.io/) reconciles topics declared as `KafkaTopic` resources, and [Terraform providers](https://www.conduktor.io/blog/kafka-terraform-providers) can manage topics and, in some cases, schemas. What they don't record is who owns a prefix, so ownership stays a convention in the repository layout.

## Conduktor Console manages both as owned resources

We built this into Conduktor Console. Topics and subjects are both [resources](https://docs.conduktor.io/guide/reference/kafka-reference) you apply with the Conduktor CLI, so the two can sit in the same file:

```yaml
---
apiVersion: kafka/v2
kind: Topic
metadata:
  cluster: prod
  name: payments.refunds
spec:
  replicationFactor: 3
  partitions: 6
  configs:
    retention.ms: '604800000'
    min.insync.replicas: '2'
---
apiVersion: kafka/v2
kind: Subject
metadata:
  cluster: prod
  name: payments.refunds-value
spec:
  schemaFile: schemas/refunds.avsc
  format: AVRO
  compatibility: FORWARD_TRANSITIVE
```

The `Topic` goes to the brokers and the `Subject` to the registry connected to the same cluster. `schemaFile` points at the Avro file in the repository, so the schema is reviewed like code, and `compatibility` is set on the subject instead of inherited from the global default.

Ownership comes from [federated ownership](https://www.conduktor.io/federated-ownership). A platform team declares an application instance that owns the `payments.` prefix for topics, consumer groups and subjects alike. Anything outside the prefix is rejected before other checks run, and no two instances can claim overlapping prefixes on a cluster.

The platform's rules are [resource policies](https://docs.conduktor.io/guide/reference/self-service-reference), written as CEL conditions:

- **Topic policies** can require a replication factor of 3, keep `retention.ms` inside a range, or enforce a naming pattern.
- **Subject policies** can allow only Avro or Protobuf, or require a specific compatibility mode.
- **`conduktor apply --dry-run`** checks the topic against Kafka with `validateOnly` and the subject against the registry's compatibility API, so a CI job that runs it fails the pull request before anything reaches the cluster.

In the UI, each topic's **Schema** tab shows its key and value subjects, and the Schema Registry view compares versions and checks compatibility before an update.

> **What it doesn't do.** The **Schema** tab finds subjects through `TopicNameStrategy`, so subjects named after a record type don't show up against the topic. Admin API keys skip resource policies by design, so keep them out of application pipelines. And a policy can't compare a resource with its previous version, so set a fixed compatibility mode per environment instead of a rule like "only stricter".

## Where to start

You don't have to move every topic at once. An order that works:

1. **Pick one naming strategy and write it down.** `TopicNameStrategy` keeps the link between a topic and its subjects visible, and it's what most tooling expects.
2. **Give every prefix an owner.** The team that owns `payments.` topics should own `payments.` subjects too.
3. **Put schemas next to the topics.** Review a schema change in the same pull request as the topic it describes.
4. **Set compatibility per subject.** Don't rely on the registry-wide default for production subjects.
5. **Turn off auto-registration in production.** Keep it for development, and let production schemas arrive through the pipeline.
6. **Then clean up the drift.** List topics with no schema and subjects with no topic, and decide for each one whether it should still exist. Console's [governance Insights](https://docs.conduktor.io/guide/insights/governance) shows how many topics have a registered schema, which is a good place to begin.

Topics and schemas are hard to manage together because Kafka and the schema registry don't know about each other, and the only link between them is a subject name. Managing them easily means giving each topic and its schemas one owner, one place in the repository and one check before anything is applied, so a topic and the shape of its data change in the same reviewed step.

[See how federated ownership works in Console →](https://www.conduktor.io/federated-ownership)

---

Related: [Kafka Schema Registry: Open by Default →](https://www.conduktor.io/blog/kafka-schema-registry-open-by-default) · [How to Manage Kafka ACLs and Security at Scale →](https://www.conduktor.io/blog/how-to-manage-kafka-acls-at-scale) · [Terraform Your Whole Kafka Platform →](https://www.conduktor.io/blog/kafka-terraform-providers)