How to Find the Producers and Consumers of a Kafka Topic

"Which applications are producing to this topic, and which ones are consuming from it?"

Most platform teams get this question, usually right before a topic is retired, migrated or given a new schema. The answer is easy for one side and hard for the other.

Consumers are easy. Every consumer joins a group, the broker stores the group's offsets, and one command lists who is subscribed. Producers are the tricky bit. Kafka stores nothing about them, so a producer that connects, writes and disconnects leaves no trace beyond the bytes it appended. Finding producers means working from ACLs, client IDs, broker metrics, tracing or a proxy in the path, and each of those tells you something different:

Evidence	Answers	For	Misses
Consumer groups	Who is subscribed right now	Consumers	Producers, idle consumers, manual assignment
ACLs	Who is allowed to read or write	Both	Whether they actually do; anything behind a shared principal
client.id and naming	Which client a connection belongs to	Both	Anyone who left the default
Broker logs and metrics	Who sent bytes, per client ID	Both	Retention; nobody keeps them; cloud brokers hide them
Tracing headers	Which service produced or consumed a message	Both	Every app that isn't instrumented
Code and config search	Who was written to touch the topic	Both	Dynamic topic names, things deployed by hand
A proxy in the path	Every produce and fetch, with the authenticated principal	Both	Nothing, but it has to be in the path

Start with consumer groups

Every consumer that calls subscribe() joins a group, and the broker stores the group's committed offsets in __consumer_offsets. One command lists them:

kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --all-groups | grep -E '^GROUP|sample_data'

Each row is a group, a partition, its lag, and the CONSUMER-ID, HOST and CLIENT-ID of the member holding it.

GROUP                  TOPIC        PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                            HOST         CLIENT-ID
billing-service        sample_data  0          33347671        33347673        2    billing-service-a70569a0-8821-4aba-8bdb-e975f1881403   /10.0.9.77   billing-service
billing-service        sample_data  1          33338535        33338536        1    billing-service-a70569a0-8821-4aba-8bdb-e975f1881403   /10.0.9.77   billing-service
...
console-consumer-15179 sample_data  0          -               33347673        -    console-consumer-d10a9ce8-6a40-49a8-9634-0339caa220ba  /10.0.9.77   console-consumer
console-consumer-15179 sample_data  1          -               33338536        -    console-consumer-d10a9ce8-6a40-49a8-9634-0339caa220ba  /10.0.9.77   console-consumer

Real output, trimmed, with two consumers attached. The first group is named after its application. The second is a laptop running a console consumer, and its - offsets mean it never committed: close the terminal and it's gone from the list.

What it can't tell you:

• Idle groups disappear. Committed offsets expire seven days after a group empties (offsets.retention.minutes). A consumer that runs monthly isn't in the list.
• Manual assignment doesn't register. A consumer using assign() has no group membership, and if it never commits, the broker has no record of it.
• Names are only as good as whoever set them. console-consumer-73912 means someone ran a CLI.
• Nothing about producers.

ACLs tell you who may, not who does

With an authorizer enabled, the ACLs on a topic name every principal allowed to touch it:

kafka-acls.sh --bootstrap-server broker:9092 --list --topic sample_data

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=sample_data, patternType=LITERAL)`:
	(principal=User:4358148, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:4358148, host=*, operation=WRITE, permissionType=ALLOW)

READ grants are candidate consumers, WRITE grants candidate producers. Here one principal holds both, and it's a number: on Confluent Cloud a service account ID you look up in their console, on a self-managed cluster the SASL username or certificate name. ACLs are the only record Kafka keeps about writers at all, which is why Stream Lineage starts from them.

Where they mislead:

• Permission isn't activity. A principal with WRITE may have produced yesterday, last year or never. An unused ACL looks identical to one in daily use.
• Shared principals hide everything. Ten applications authenticating as svc-kafka show up as svc-kafka. Kerberos setups end up here often.
• Prefixes and wildcards widen the list. A WRITE grant on sample_ or on every topic names teams who could write here and never will.
• Superusers and defaults don't show. super.users bypass ACLs, and allow.everyone.if.no.acl.found=true makes a topic with no ACLs writable by anyone.
• Managed clusters keep some of it elsewhere. MSK with IAM auth stores permissions in IAM policies. Confluent Cloud RBAC role bindings aren't in this output.

Make clients name themselves

Every Kafka client sends a client.id with each request, and it shows up in consumer group output, broker logs and per-client metrics. If every team set it to their application name, most of the detective work disappears.

Most don't. The defaults are producer-1 and consumer- plus the group ID and a counter, so an unattended cluster fills with anonymous clients. The broker accepts any value, including an empty string.

Naming is a policy, and where you enforce it decides what it's worth:

• A wiki convention covers the teams who read it.
• A CI check on client configs covers the apps deployed through CI.
• A rule on the wire covers every client, including the laptop running a console producer. Kafka has no hook for this; a proxy does.

Broker logs and metrics

The broker sees every request, produces included, and keeps a little of it in three places:

Source	What it records	The catch
Request logger (kafka.request.logger at DEBUG)	Every request: API key, client ID, topic, partitions, authenticated principal	Gigabytes an hour on a busy broker, so nobody leaves it on
Quota metrics (kafka.server:type=Produce,client-id=…, and Fetch)	Produce and fetch byte rates per client ID and per user	Per client, not per topic: a producer on three topics is one number
Authorizer logs (kafka.authorizer.logger; Confluent Platform audit logs)	Allow and deny decisions per request, including produce and fetch	Denials only by default; allows at DEBUG cost the same as the request logger

On a managed cluster none of this is yours to switch on. MSK exposes broker and topic metrics, not per-client produce activity. Confluent Cloud exposes Stream Lineage rather than broker logs.

Tracing, if every app is instrumented

Instrumented clients carry trace context in message headers, and the OpenTelemetry messaging conventions record the topic and consumer group on each span. A tracing backend turns that into a service map: which services produce to and consume from each topic, with latency attached.

It's the only method that gets producer identity from inside the message, and the only one that requires changing every application:

• Uninstrumented clients are invisible. Connectors, third-party tools, a Python script from another team.
• It only looks forward. Tracing shows who touched the topic since you turned it on, not who has written to it for three years.
• Sampling drops rare writers. At one percent, a producer sending one message a day may never appear.

Grep the repos

Search every repository for the topic name, plus Connect configs, ksqlDB and Flink statements and Terraform. That finds intent: everything written to touch the topic.

$ rg -l 'sample_data' --glob '!*.md'
orders-ingest/src/main/java/com/acme/orders/OrdersProducer.java
billing-service/src/main/resources/application.yml
connectors/s3-sink-orders.json
analytics/flink/large_orders.sql
infra/terraform/kafka/topics.tf

Five hits, four teams. The Flink statement and the S3 sink are the ones nobody remembers when asked who reads the topic.

It misses topic names built at runtime, anything outside the repositories you can see, and anything started by hand. It also finds code deployed once and never run again. A good cross-check, weak alone.

Put a proxy in the path

A Kafka proxy terminates every client connection, so it sees every produce and fetch with the authenticated principal, the client ID and the topic, with no change to the applications. That closes the producer gap for every client, not only the instrumented ones.

It's also where naming gets enforced: a proxy can reject requests whose client.id doesn't match a pattern, or rewrite the ID on the way through.

The cost is being in the path. Clients connect to it instead of the brokers, a per-application change, and Confluent Cloud's fully managed connectors and Flink never pass through it.

Permitted, subscribed, actual

You want to know	Ask	Trust it for
Who may read or write	ACLs	The complete list of candidates, if principals aren't shared
Who is subscribed	Consumer groups	Active consumers, this week
Who does read or write	Traffic, from a proxy or instrumented clients	What is happening now, for the clients it can see

Lineage tools combine those sources into one graph. Confluent's Stream Lineage builds its graph from the last ten minutes of client activity, on Confluent Cloud clusters and on Confluent Platform clusters registered with it. DataHub and OpenMetadata build theirs from declared metadata and connector configs.

At Conduktor we started from the permitted side, because ACLs are the one record every Kafka cluster already has. Console's Stream Lineage draws the graph from the ACLs, attaches the owners declared in Self-service, and on clusters behind Gateway lays live throughput over each connection. Permitted, subscribed and actual, on one screen:

Stream Lineage with Live switched on. The edges come from ACLs, the thickness from Gateway traffic, and an idle grant stops looking like an active consumer.

If you want to see the combined view on a real topic, Stream Lineage Is Useless. So Why Did We Build It? walks through retiring one, and the Stream Lineage docs cover setup. Otherwise we hope this was useful, and that it gives you a reference for working out who produces and who consumes on your own cluster.

---

Related: Kafka Access Management → · Kafka Consumer Groups Explained → · Kafka ACLs and Authorization Patterns →