# Every Kafka Lineage Tool Reads a Different Source

Point two lineage tools at the same Kafka topic and they can draw two different graphs. A data catalog shows the S3 sink connector and the Flink job, because someone ingested their configs. A tracing tool shows the two services that run a tracer. Neither shows the Python script another team runs against the topic every night.

Both graphs are correct for what they read. Kafka doesn't store a record of who produces to a topic, so every lineage tool builds its edges from some other evidence, and the evidence decides which applications you see. Picking a [data lineage](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption) tool for Kafka means picking that evidence first.

How the main options compare:

| Tool | Edges come from | Sees a producer with nothing added to the app | Permitted or active | Team ownership | Covers systems beyond Kafka | Runs against |
|---|---|---|---|---|---|---|
| [Confluent Stream Lineage](https://docs.confluent.io/cloud/current/stream-governance/stream-lineage.html) (Confluent Cloud) | Client activity seen by Confluent's brokers | Yes | Active, recent window | — | Confluent connectors, ksqlDB, Flink | Confluent Cloud clusters |
| [DataHub](https://datahubproject.io/docs/generated/ingestion/sources/kafka/) (Apache 2.0, or DataHub Cloud) | Ingested metadata: topics, schemas, Kafka Connect configs, emitted events | No | Declared | Owners set in the catalog | Warehouses, BI, orchestrators | Any Kafka |
| [OpenMetadata](https://docs.open-metadata.org/connectors/messaging/kafka) (Apache 2.0, or Collate) | Ingested metadata, pipeline connectors, edits in the UI | No | Declared | Owners set in the catalog | Warehouses, BI, orchestrators | Any Kafka |
| [OpenLineage](https://openlineage.io/) + [Marquez](https://marquezproject.ai/) (Apache 2.0) | Run events emitted by jobs (Flink, Spark, Airflow) | No | Active, per job run | — | Every job that emits events | Any Kafka |
| [Datadog Data Streams Monitoring](https://docs.datadoghq.com/data_streams/) (SaaS) | Datadog tracers inside each service | No, each service runs the tracer (an agent can attach it without code changes) | Active | Service and team tags | Other queues the tracer supports | Any Kafka |
| OpenTelemetry + a tracing backend (Apache 2.0 SDKs) | Spans from instrumented clients | No, each service runs an SDK or agent | Active, sampled | Service name | Anything instrumented | Any Kafka |
| [Conduktor Console Stream Lineage](https://docs.conduktor.io/guide/conduktor-concepts/stream-lineage) (commercial, paid plans) | ACLs or Confluent role bindings; throughput from Gateway | Its principal, if it holds a `WRITE` grant on the topic | Both: permitted from ACLs, recent rates on Gateway clusters | Application and team from the ownership model | No | Any Kafka with ACLs; role bindings on Confluent |

Five kinds of evidence a lineage tool can read. Each one leaves a different set of applications off the graph.

## Plain producers leave no metadata

Take a Java service that writes to `orders` with the standard Kafka client and no tracer. The broker accepts the records and keeps no record of the writer beyond the bytes. A lineage tool can only find this producer if it reads evidence the service creates without anyone changing it.

Two kinds of evidence qualify. The broker sees the connection, so a tool that reads broker-side activity sees the producer. And if the cluster runs an authorizer, the service needs a `WRITE` [ACL](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) on the topic, so a tool that reads ACLs sees the principal.

The other sources need someone to add something:

- **Catalog ingestion.** DataHub and OpenMetadata read topics, schemas and connector configs from the cluster and Kafka Connect. A plain producer appears only if someone emits lineage for it through the catalog's API or draws the edge by hand.
- **Job run events.** OpenLineage records the Kafka datasets a Flink or Spark job reads and writes, through an integration in that framework. A microservice emits nothing unless a team adds the OpenLineage client to it.
- **Client tracers.** Datadog Data Streams Monitoring and OpenTelemetry need a tracer in each service, added to the code or attached by an agent at deploy time. Every connector, third-party tool and script without one stays off the map.

For the question platform teams get most, "who writes to this topic", the evidence that needs no code change wins. A catalog answers a different question well: where a dataset goes after it leaves Kafka.

## Permitted and active are different edges

An ACL says a principal may read a topic. It doesn't say the principal read anything this month. Traffic says a client is reading now, and says nothing about a principal that holds a grant and reads once a quarter.

Each source lands on one side:

| Source | Edge means | Wrong when |
|---|---|---|
| ACLs or role bindings | This principal may read or write | The grant is unused, or ten applications share one principal |
| Broker activity | This client produced or fetched recently | The client runs less often than the tool's window |
| Tracers and job events | This instrumented service or job touched the topic | The client isn't instrumented, or sampling dropped it |
| Catalog metadata | Someone declared or ingested this edge | Nobody updated it after the change |

Retiring a topic needs both sides. The permitted list tells you who to warn, including the monthly batch job. The active list tells you whether anything still reads, so an idle grant can be revoked instead of negotiated.

Confluent Stream Lineage draws from recent client activity on Confluent Cloud, which makes it strong on the active side and quiet about a consumer that hasn't run inside that window. Conduktor Console draws its edges from ACLs, and on clusters behind Conduktor Gateway its Live toggle lays throughput over each edge, averaged over the observability Interceptor's flush interval, so an idle grant shows as a thin grey line.

## A principal name doesn't say which team to call

Every source above ends at an identifier: a principal such as `User:sa-4358148`, a client ID, a service name in a tracer, a job name. Before retiring a topic you need a person, and Kafka doesn't store which team owns a principal.

The tools solve it in three ways:

- **A catalog owner field.** DataHub and OpenMetadata let you set an owner on each dataset and pipeline. It's as current as the last person who edited it.
- **Tags on the service.** Datadog groups by the `service` and team tags your deployments set, which works when every team sets them the same way.
- **An ownership model that also grants access.** In Conduktor Console, an application owns its service accounts and topics, and the [federated ownership](https://www.conduktor.io/federated-ownership) model generates the ACLs from those declarations. The lineage node carries the application and team because the same record that names the owner created the grant.

The last approach only covers what you've registered. Console's Service account view lists every principal with ACLs, registered or not, which is where an unowned grant shows up.

## Lineage that continues past Kafka

Some questions don't stop at the topic. A compliance team asking where a customer's email ends up needs the S3 bucket, the warehouse table and the dashboard built on it.

Catalogs are built for this. DataHub and OpenMetadata ingest from warehouses, BI tools and orchestrators as well as Kafka, and join them on one graph. OpenLineage covers the same ground from the job side: every Spark, Flink or Airflow run that emits events adds its inputs and outputs, whatever they are.

Confluent Stream Lineage includes the connectors, ksqlDB queries and Flink statements that run inside Confluent Cloud. Conduktor's Stream Lineage stays on Kafka: applications, service accounts and topics. A team that needs both often runs a catalog for the cross-system view and a Kafka-side tool for producers and consumers.

## Matching a tool to the cluster you run

Start from the evidence your clusters already produce:

- **Everything runs on Confluent Cloud.** Stream Lineage is part of the platform and reads client activity without changes to your applications. It covers Confluent Cloud clusters, so plan separately for anything self-managed or on another provider.
- **You need the warehouse and the dashboards on the same graph.** Run a catalog such as DataHub or OpenMetadata. Plain producers appear only once your services emit lineage.
- **Most of your streaming work is Flink and Spark jobs.** OpenLineage, with Marquez or a catalog that accepts its events. It records each run's inputs and outputs precisely, and only for jobs.
- **Every service already runs Datadog APM.** Data Streams Monitoring adds producer-to-consumer maps with latency on each hop. Anything without the tracer is missing.
- **You run OpenTelemetry across your services.** The messaging spans already carry topic and consumer group, and a tracing backend's service graph draws the map.
- **Clusters on several providers, ACLs on all of them, and teams that need to know who owns what.** [Conduktor Console](https://www.conduktor.io/console) draws lineage from the ACLs, or role bindings on Confluent clusters, on every cluster it manages, labels each node with its application and owning team, and adds live throughput on clusters behind [Gateway](https://www.conduktor.io/gateway). Stream Lineage is in Console 1.46 and later on paid plans, and the Live layer arrived in 1.47.

## What none of these record

The gaps that remain whichever tool you pick:

- **Field-level lineage inside Kafka.** None of the Kafka-side tools read message contents to tell you that `email` in one topic became `contact` in another. Catalogs can show column-level lineage for jobs that report it.
- **A shared principal split back into applications.** When ten services authenticate as `svc-kafka`, ACLs and broker activity show one principal. Only client IDs or tracers can tell them apart, and only if teams set them.
- **History by default.** Console's graph is current state, and activity-based tools keep a limited window. Proving who read a topic last year takes audit logs kept for that purpose.
- **Clients that bypass the evidence.** A managed connector outside the proxy has no live rates in Console. A client on a cluster your tool doesn't watch appears nowhere.

A Kafka lineage graph shows exactly the applications its evidence can see. Decide first which applications you can't afford to miss when a topic changes, then pick the tool that reads the evidence those applications leave. The same question comes back every time a topic is retired, a schema changes or a cluster is migrated.

## Questions about Kafka data lineage tools

**How do I find which applications consume a Kafka topic?**

Run `kafka-consumer-groups.sh --describe --all-groups` and filter on the topic for active consumer groups. Consumers that use `assign()` without a group don't appear there. Then list the topic's ACLs for every principal allowed to read it, including consumers that only run occasionally. A lineage tool combines the two on one graph.

**How do I find which applications produce to a Kafka topic?**

Kafka doesn't record producers. Look at `WRITE` ACLs on the topic, client IDs in broker request logs or quota metrics, tracing data if services are instrumented, or traffic seen by a proxy in front of the cluster.

**Does Kafka have built-in lineage?**

No. Apache Kafka stores consumer group offsets and ACLs, which are the raw material for lineage, but no graph of producers and consumers.

---

Related: [How to Find the Producers and Consumers of a Kafka Topic →](https://www.conduktor.io/blog/who-produces-and-consumes-a-kafka-topic) · [Stream Lineage Is Useless. So Why Did We Build It? →](https://www.conduktor.io/blog/stream-lineage-is-useless-so-why-did-we-build-it)
