Kafka Topic Naming Conventions That Scale

Stéphane Derosiaux July 13, 2024 2 min read
Kafka Topic Naming Conventions That Scale

Topic naming feels like bikeshedding until you have 500 topics and can't find anything. The convention you choose becomes permanent infrastructure: Kafka topics cannot be renamed.

I've seen organizations stuck with OrderEvents, order_events, order-events, and ORDEREVENTS across 20 teams. Nobody can find anything. ACLs require one entry per topic.

We started with ad-hoc names. Five years later, we have 800 topics and no consistent way to search or apply ACLs. A proper naming convention would have saved hundreds of hours.

Platform Lead at a logistics company

What does this post change for your team?ClaudeChatGPT

A Pattern That Works

The most battle-tested pattern uses hierarchical segments:

<domain>.<entity>.<event>

Examples:

orders.order.created
orders.order.cancelled
inventory.item.reserved
payments.transaction.completed

This enables wildcard ACLs:

# Grant service access to all order topics with one rule
kafka-acls.sh --bootstrap-server localhost:9092 \
  --add --allow-principal User:order-service \
  --operation Read --operation Describe \
  --topic orders. --resource-pattern-type prefixed

At 500 topics, consistent prefixes mean ~10 ACL entries instead of 500.

Pick One Separator

Kafka warns about mixing . and _ because they collide in metric names:

kafka-topics.sh --create --topic "test.naming" ...
# WARNING: topics with '.' or '_' could collide

Pick one and enforce it. Most organizations choose . for hierarchy, - for word separation: orders.order-created.

Aligning With ACLs

Naming PatternACL Complexity
orders.order.created1 prefix ACL per domain
OrderCreated, order_created1 ACL per topic

Common Mistakes

Naming after teams:

checkout-team.order-events  # Bad
orders.order.created        # Good

Teams rename. Services get rewritten. The topic stays forever with a misleading name.

Naming after consumers:

analytics-orders            # Bad

A topic describes data, not who reads it.

Environment prefixes:

prod.orders.order.created   # Bad

Use separate clusters for environments instead.

Enforcing Conventions

# server.properties - Force controlled topic creation
auto.create.topics.enable=false

Beyond configuration, a self-service portal can enforce naming conventions automatically when teams request new topics.

Define topics in source control and validate in CI:

# topics.yaml
topics:
  - name: orders.order.created
    partitions: 6
    replication: 3

Your pipeline validates name matches ^[a-z]+\.[a-z-]+\.[a-z]+$.

Kafka Streams Internal Topics

Streams creates internal topics automatically:

my-app-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog

If you change your topology, these names change. Fix: Name operators explicitly:

stream.groupByKey(Grouped.as("orders-by-customer"))
      .aggregate(..., Materialized.as("order-counts"));

The organizations running Kafka most efficiently invested in naming conventions early. The ones struggling grew into whatever pattern emerged from ad-hoc decisions.

Book a demo to see how Conduktor Console provides centralized topic governance with naming validation and policy enforcement.