How do you make your Kafka ecosystem respect GDPR or DORA? If the answer depends on every producer and consumer getting it right, often hundreds of them, across many languages and varying levels of maturity, it depends on too much. I want to argue the opposite.
Privacy in Kafka is an ownership problem before it is an implementation problem. It belongs in one enforcement layer with a single owner, not scattered across a hundred applications.
DORA asks EU financial entities to demonstrate control over the ICT systems their critical functions depend on, and Kafka is often one of them: who can access it, how the data is protected, what you can put in front of a supervisor. The obligation sits with the entity, but in practice it lands on whoever happens to be writing a producer or a consumer.
A control that lives in a hundred codebases is a control nobody can evidence.
Kafka is not a temporary buffer
🚫 "The data is temporary anyway, Kafka is just a buffer."
Plenty of Kafka data never expires. Compacted topics keep the latest value for every key indefinitely: that is how stream processing holds its state, and how a lot of teams end up running Kafka as the system of record rather than a buffer. Then think about the data flow end-to-end:
- the replica in another region
- the Connect sink into a warehouse (yes, and the analytics copy someone exported last time to run a local DuckDB)
- the third party you share data with
If it's personal data, it's yours to track and protect.
Kafka was not built for any of this. It was built for throughput and durability. Its core promise, an append-only immutable log, is at odds with much of what privacy law assumes you can do.
Kafka lacks privacy by design
Major privacy laws (GDPR, CCPA/CPRA) ask for the same things, and each hits the same wall: Kafka is append-only and immutable by design (no SQL DELETE!).
| What a person can ask you | Why Kafka makes this hard |
|---|---|
| Keep their data protected | Any authorized consumer reads the plaintext and can do anything with it |
| Delete their data (the right to be forgotten) | You cannot delete one record from an immutable log |
| Show them their data (the right of access) | No catalog, no lineage, no query: the log is a stream, not a queryable store |
| Fix data that is wrong (the right to rectification) | You cannot edit a record in place; correction means appending a new event |
| Use it only for what they agreed to (consent and purpose) | Kafka ACLs are coarse, no notion of who can read what or who reads what for what |
Privacy by design means protection built in by default, not bolted onto hundreds of uncontrolled, unaudited applications. A single enforcement layer is the cleanest way to get there.
Building privacy in every app? Here be dragons.
Say ownership is settled and the security team owns the rule (that shift has its own post). The mechanism still has to live somewhere.
Protecting the data comes down to two moves: encrypt the sensitive fields in Kafka itself, and control who can read which fields. How do you encrypt PII in Kafka? Via a homemade client-side library or Confluent CSFLE? Yes, both can ship fast, it's just code after all. But let's go deeper:
- The compatibility matrix explodes. Building an encryption helper is easy. Then come Avro, JSON and Protobuf; then Vault, AWS, Azure, GCP and Thales KMS; then Java, Python, Go, Rust and Node; then nested fields and key rotation. What started as a helper becomes a full-time project: a roadmap, internal advocacy, and an ever-growing test suite.
- Key management governance. Every service now needs access to your KMS, which means holding credentials. Every language does crypto its own way. Field sensitivity changes over time. The result is many inconsistent points of failure.
- No enforcement. Enforcement is opt-in: each app has to build and configure it correctly, and it often gets disabled in dev. If an app forgets or misconfigures it, plaintext lands in a topic that was supposed to be encrypted.
- Retention-only does not work. Setting short retention feels like deletion, but it does not erase the replica, the backup, the dead-letter topic, or the copy a connector already landed downstream. And on a compacted topic, there is nothing to expire in the first place.
- Downstream cannot be trusted. Once data leaves Kafka through all your integrations and pipelines, how do you even track it, let alone protect it?
That's quite depressing.
The hard part of privacy in Kafka is answering: "where is the rule, who owns it, is it applied consistently, and what happens when the CISO wants to change it?" The do-it-yourself path spreads privacy across every team until it belongs to no one you can name, and that is the governance liability. See field-level encryption without the DIY tech debt.
Compliance belongs in one layer, not one hundred apps
Once the network only lets clients reach Kafka through the Gateway, enforcement cannot be skipped, and privacy enforcement gets a single accountable owner. The security team defines the rule once; developers ship their usual code; the rule applies to all traffic without any redeploy, the same ownership shift a Kafka data platform needs.
That layer is a proxy sitting between applications and Kafka. Like a web proxy, clients point at the proxy's address instead of the broker's. No application code changes. Policies run at the proxy: payload or field-level encryption, data masking, and access rules, applied to every client, in any language.
It runs the same whether you have MSK, Confluent, Aiven, Redpanda or open-source Kafka clusters, as long as client traffic is routed through the Gateway. You can even bring your own encryption to managed Kafka.
Bringing data privacy to Kafka
Let's take each right two ways: in-app, where every team carries it, then in-proxy, where the layer enforces it once.
Keep their data protected
In-app
Every team integrates a client-side crypto library or a custom serializer, then maintains it across languages, KMS backends and CVEs, hands KMS credentials to every service, and hopes no app skips the step.
In-proxy
Field-level encryption runs at the Gateway. Producers send normal JSON, Avro or Protobuf; the Gateway encrypts the whole payload or just the named fields before the broker ever sees them. Only the Gateway needs access to the KMS (Vault, AWS KMS, Azure Key Vault, GCP Cloud KMS or Fortanix DSM). This is field-level encryption with zero code changes, enforced at the Gateway that every client connects through, not opt-in per app, which is what DORA asks for when it ties encryption to your data classification rather than to each team's discipline.
The right to be forgotten
In-app
Kafka's append-only log makes physical deletion almost impossible, and never within a deadline, once you count the replicas, backups, dead-letter topics and connectors moving data out of sight.
In-proxy
Crypto shredding for GDPR erasure builds on field-level encryption. You scope each subject's encryption to their own key (via key-ID templating). On an erasure request, you delete that one key from the KMS, and every record for that subject becomes permanently unreadable.
Crypto shredding erases data you encrypted under a key. If a copy was already decrypted into a downstream store, erasing it there is that store's problem.
The right to see their data
In-app
Someone has to locate every topic, consumer group, connector and sink where a subject's data flows, then assemble it.
In-proxy
Conduktor provides the discovery substrate: Console Application resources bundle topics, consumer groups, subjects and connectors with owner, purpose, data categories and retention; lineage shows the real producer-to-consumer flows; audit logs record who read what. That solves the hard part of a data-access request, finding where the data is and who touches it, and it doubles as the asset and dependency map DORA expects you to keep.
The right to fix wrong data
In-app
With Kafka, you cannot edit a record in place. Correction means appending a corrected event or relying on log compaction by business key, then making every downstream consumer update its state. Consumers are often inconsistent in how they handle this, leaving stale data here and there.
In-proxy
Conduktor adds guardrails and visibility: data-quality and schema-validation policies enforce that corrected events are well-formed and follow the contract before they are accepted; lineage shows which downstream consumers must reprocess; and access control governs who may publish a correction.
Consent and purpose
In-app
This is about purpose limitation: "data collected for A may not be reused for B." It usually isn't enforced, because it's too complex to handle across every Kafka app. Broker ACLs are far too coarse, and they never read the payload, so they are blind to purpose and to who consumes what.
In-proxy
Consent state lives in your consent system. The Gateway enforces its downstream consequence, who may read which fields, and that is where it is strong:
- Per-reader masking and per-consumer decryption give the same physical topic different visibility per consumer: marketing sees company and industry while PII stays encrypted, fraud sees the full record. That is data minimization enforced at read time, without duplicated topics (one source of truth).
- Virtual clusters isolate by identity. A consumer authorized for one purpose cannot see or list another's topics; Partner Zones fence off external recipients. Isolate non-prod Kafka with virtual clusters on one physical cluster, each with its own security rules.
- Application resources make purpose a declared attribute of each app, so permissions derive from the declaration rather than manual grants.
- Audit logs record who consumed what, as evidence to reconcile against declared purpose, or to hand to a supervisor asking how access is kept to the minimum under DORA.
Conclusion
None of this makes you "compliant" on its own. It supports the obligations and removes the per-app coordination problem that makes them fragile.
The real decision is who owns privacy. In a hundred applications, it depends on every developer getting security right, forever. In a proxy layer, it is one policy the security team owns and changes at will, without a redeploy.
The same encryption, masking, isolation and audit controls that answer GDPR are the ones behind one control layer for DORA, GDPR and IT-Grundschutz. You build the layer once, not three workstreams.
If you are mapping your own requirements onto Kafka, talk to us.