Most Kafka Data Governance Tools Never Read a Record

A search for the best Kafka data governance platform returns a mixed list: a schema registry, a data catalog, a proxy, a Terraform provider. They rarely compete with each other, because each one checks something different at a different moment.

That moment decides what a tool can stop. A check that runs when a schema is registered can't see the records written afterwards. A check inside a producer's serializer runs only in the producers that use that serializer. A check after the write can report a bad record, but the record is already in the topic, and consumers read it.

Most of these tools never look inside a record at all. So for each tool, two facts matter most: where it sits relative to the write, and whether a producer can skip it.

Kafka governance and data quality tools compared

The tools most often named for Kafka governance and data quality, by where they check:

Tool	Where it checks	Reads record content	Can a producer skip it	License
Kafka ACLs and CreateTopicPolicy	Broker, on admin requests	No	No	Apache 2.0
Strimzi or Terraform	Provisioning pipeline	No	Yes, through the Admin API	Open source
Confluent Schema Registry, Apicurio, Karapace	Schema registration	No	Yes, by not using the registry	Community License (Confluent), Apache 2.0
Confluent data contract rules	Client serializer	Yes, CEL rules	Yes, with another serializer	Commercial
Confluent broker-side schema ID validation	Broker	Schema ID only	No	Commercial
Redpanda schema ID validation	Broker	Schema ID only	No	Enterprise
Kroxylicious record validation	Proxy	JSON syntax, JSON Schema	Only by bypassing the proxy	Apache 2.0
DataHub, OpenMetadata	Catalog, after the write	Metadata only	—	Apache 2.0
Conduktor Gateway and Console	Proxy, provisioning	Yes: CEL, JSON Schema, schema payload	Only by bypassing Gateway	Commercial

The two broker rows are easy to misread. The Confluent and Redpanda features confirm that a record carries a schema ID the registry knows, without deserializing the payload, so a record with a valid ID and a negative amount is written.

Catalogs sit after the write. DataHub and OpenMetadata ingest topics, schemas and owners from Kafka and the registry so people can find data and its owner, and any problem they show is already in the topic.

Kafka's own hooks govern topics and ACLs

Before a record exists, governance is about resources: who may create a topic, with which name, partition count and retention, and who owns it afterwards. Kafka has two built-in hooks for this. ACLs decide whether a principal may create topics under a prefix. A broker plugin named in create.topic.policy.class.name can reject a topic request based on its settings.

Both work within limits. An ACL answers yes or no, so it can't say "only with a team. prefix and a data-classification label". A policy plugin is Java deployed on every broker, and many managed Kafka services don't let you load one. Kafka also has no place to store an owner or a label on a topic, so the answer to "who owns payments.v2" lives in a wiki or a spreadsheet.

GitOps tools move the decision into code review: Strimzi's Topic Operator reconciles KafkaTopic resources, and the Terraform providers do the same from HCL. A pipeline governs only what goes through it, so a Kafka Streams app creating internal topics still talks to the cluster directly. None of these hooks looks at records: once a principal may write to a topic, Kafka appends whatever bytes it sends.

A schema registry checks each version once

A schema registry stores schemas and gives each version an ID that serializers write into every record. When a producer registers a new version, the registry compares it with earlier versions under the subject's compatibility mode. With BACKWARD, the default in Confluent Schema Registry, a new Avro version that adds a field without a default is rejected, because a consumer on the new version couldn't read older records.

Three limits follow from where this check sits:

• It judges shape. The registry confirms that amount is a number. It has no opinion on whether the number is positive, in cents, or filled in at all.
• It runs only for clients that use it. A producer writing JSON with a StringSerializer never contacts the registry, and the broker doesn't notice.
• It runs at registration time. Once a version is accepted, the registry isn't consulted again about the records written with it.

Build plugins such as the Confluent Maven plugin's test-compatibility goal run the same check in CI, for the producers your pipeline builds.

Client rules run wherever the producer runs

Confluent's data contracts attach rules to a schema. A domain rule is a CEL expression such as message.amount > 0, and the serializer evaluates it before the record leaves the producer. When the rule fails, its action either fails the send or writes the record to a dead letter topic.

This is the first tool in the list that checks content, and it runs before anything crosses the network. Rule support depends on the serializer library and its language, with Java the most complete. A producer on another serializer, an older client version or a non-Confluent library skips the rules without any error.

In the conversations we have with platform teams, data quality standards are often written for batch pipelines and say little about streaming. Client-side rules suit a team that owns every producer and its build. On a shared cluster with many teams and several languages, the platform team usually owns neither.

Only the write path sees every producer

A check between the producer and the log can't be skipped by a client, because every produce request crosses it. Each tool sits at one point on a record's way into a topic:

Broker-side validation runs inside the broker. On Confluent Server, setting confluent.value.schema.validation=true on a topic makes the broker reject records whose schema ID is missing or unknown to the registry. Redpanda has the same check through the redpanda.value.schema.id.validation topic property. Neither deserializes the payload, so the check proves that a record names a registered schema. It doesn't prove the bytes match that schema.

Open-source Apache Kafka has no broker-side record validation. KIP-729 proposed custom validation before log append, and it hasn't shipped.

A Kafka proxy sits between clients and brokers and speaks the Kafka protocol on both sides. Because it receives the full produce request, it can deserialize each record, check it against a schema or a rule, and refuse the request before the broker writes anything. The costs are an extra network hop, CPU for every record it deserializes, and coverage limited to the clients whose bootstrap servers point at it.

What none of these tools do

Every option above leaves gaps, and Conduktor is no exception:

• Write the rules for you. Each content check encodes business knowledge, such as amount being positive or country being an ISO 3166 code. Someone who knows the data has to write each rule and keep it current.
• Repair what's already in the topic. A write-time check protects the next record. Bad records written before it still need a replay or a cleanup.
• Cover clients that bypass them. A proxy covers clients routed through it. If producers can still reach the brokers directly, network rules or ACLs have to close that path.
• Reject a single record. Gateway's block action rejects the whole record batch for that partition with a non-retriable INVALID_RECORD error. A producer that only logs send errors loses the valid records in that batch.
• Block from outside the write path. Conduktor Console without Gateway checks records after they're produced and reports violations without blocking. Console resource policies don't apply to requests made with an Admin API key.

Match the tool to who writes to your cluster

Conduktor puts content checks on the write path and resource checks at provisioning. Gateway evaluates data quality policies on every produce request: CEL rules over the value, key and headers, a JSON Schema, or a check that each record carries a registered schema ID and matches that schema. A policy can block the batch, mark violating records with a conduktor.dataquality.violations header, or copy them to a dead letter topic. In Console, resource policies apply CEL rules to topics, subjects and connectors as teams create or change them through federated ownership, and the Schema Registry Proxy adds per-subject write permissions in front of Confluent Schema Registry.

Start from who produces to your cluster and what you need to stop:

• Every producer is a Java service you build. Confluent data contract rules, plus compatibility checks in CI.
• You run Confluent Platform or Redpanda and want a floor no client skips. Turn on broker-side schema ID validation, and put content rules somewhere else, since it doesn't read the payload.
• Your problem is topic sprawl and naming. Set auto.create.topics.enable=false, manage topics with Strimzi or Terraform, and give create ACLs to the pipeline's principal only.
• You want an open-source proxy and can operate it. Kroxylicious with its record validation filter.
• People can't find data or its owner. A catalog such as DataHub or OpenMetadata.
• Many teams, several languages, and content rules that must hold for every producer. Conduktor Gateway and Console apply the rules on the write path and the topic policies at creation, from one place.

A Kafka data governance platform is defined by where its checks run. ACLs, registries, pipelines and catalogs govern resources, schemas and metadata, and none of them reads a record. Only a check on the write path applies to every producer, and of those, only a proxy can read the payload before the broker stores it.

---

Related: Kafka Data Quality: A Schema Registry Isn't Enough → · Kafka Data Contracts: A Schema Is Not a Contract → · Kafka Governance: Automated Guardrails Over Wiki Policies →