How Do You Handle PII in Kafka?

Say a team adds the customer's email address to the orders topic. From then on, every service that reads the topic gets every customer's email, whether it needs it or not.

Only the fraud service needs it. The search indexer and the analytics job don't, and neither does the test environment that gets a copy of production. The cloud provider running your brokers shouldn't be able to read it either.

So handling PII in Kafka starts with one question per field: who must never read it? The answer decides which protection you need.

Kafka Protects Topics, Not Fields

Kafka ships the controls you'd expect, and all of them are worth turning on. The security section of the Apache docs covers authentication, authorization with ACLs, and TLS for traffic between clients and brokers. None of them looks inside a record.

Here's what each native mechanism does for the email field:

Mechanism	What it protects	What it leaves readable
TLS	Records on the network	Records on the broker's disk
Disk or volume encryption	A disk that leaves the data center	Everything the broker serves to clients
ACLs	Which principals can read a topic	Every field of a topic you can read
Retention and compaction	Old segments, older values for a key	One customer's records everywhere else

The ACL row matters most for PII. An ACL grants an operation on a resource, so READ on orders returns the whole record to the fraud service, the search indexer and the analytics job alike.

This works when everyone who reads a topic is allowed to see everything in it. It stops working the day one field has a smaller audience than the topic it lives in.

You Can't Protect a Field Nobody Has Classified

Before any mechanism, somebody has to know that email is in orders at all. A question we hear a lot from platform teams is how they'd even find out which topics carry personal data.

There are two common ways to record which fields are sensitive:

• A list the platform team owns. It works on any data format, including schemaless JSON, but it goes stale when a producer adds a field and nobody tells the platform team.
• Tags in the schema. The producing team marks email as PII in its schema. It only covers topics with a schema you control, and platform teams tell us they can't count on every producer team to tag its own fields.

Most teams end up with both, plus a fallback for the topics nobody declared at all.

Decide Who Must Never Read Each Field

With the fields known, go back to orders and list the readers who shouldn't see email. Each one is stopped by a different mechanism, and each mechanism leaves something open:

Who must never read the value	Mechanism that stops them	What stays open
People browsing topics in a Kafka UI	Masking in the UI	Applications and CLI tools
A consumer app, or a test copy of production	Masking on consume	The clear value stored on the brokers
Broker operators, the cloud provider, backups	Encryption on produce	Consumers you allow to decrypt
An analytics team that counts or joins on the field	Tokenization	Places where clear values and tokens meet
Everyone, after an erasure request	A key per customer, deleted (crypto shredding)	Copies already decrypted elsewhere

In the orders example, the fraud service is the only reader that needs the clear address. The search indexer only needs masking on consume, the analytics job may need tokens if it counts customers, and the test copy needs masking before it's written. The cloud provider is only kept out by encryption, because masking never changes what the broker stores. Most teams need two or three of these at once, not the one that's easiest to turn on.

Masking Hides PII. It Doesn't Protect It.

🚫 "We mask PII in Kafka, so it's protected."

Masking replaces a value with **** as it's read. The record stored in Kafka doesn't change, so masking decides what one reader sees, not what the cluster holds. Masking in a UI is narrower again: it hides the value from people browsing topics and does nothing for applications.

Encryption works the other way round. The value is encrypted on produce, before it reaches the broker, so the stored record holds ciphertext and only the readers you allow can decrypt it:

The difference shows up with a client that connects straight to the brokers. With masking it reads the clear address, and with encryption it reads ciphertext.

Masking still has a clear job: making safe copies. Consume production through masking, produce the result into a test environment, and that copy has nothing to decrypt or reverse.

Put the Rules Between Clients and Brokers

Masking on consume, encryption on produce and per-reader decryption all need something that sees every record on its way in and out. Kafka brokers store record values as the producer sent them, so that work happens either inside every client or in front of the brokers.

Client-side encryption libraries are where plenty of teams start, but every client in every language has to adopt the same library, and one that skips it writes plaintext.

The other place is a Kafka-protocol proxy that every client connects through, so applications keep their code. Conduktor Gateway implements this with interceptors for encryption, masking, tokenization and crypto shredding. An interceptor can be scoped to a service account, a group or a Virtual Cluster, which is how per-reader access works. Here the fraud group can decrypt email on orders, and clients outside that group keep reading ciphertext:

apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: decrypt-email-for-fraud
  scope:
    group: fraud
spec:
  pluginClass: io.conduktor.gateway.interceptor.DecryptPlugin
  priority: 100
  config:
    topic: "orders"
    kmsConnectionName: PROD_VAULT
    recordValueFields:
      - email

scope.group limits the interceptor to clients in the fraud group, and topic limits it to orders. recordValueFields names the one field this group may decrypt. kmsConnectionName points at the KMS that holds the master key, which Gateway uses without ever seeing it.

Erasure Needs a Key per Customer

Kafka can't delete one customer's records on request. Retention removes data by age or size for a whole topic, and a tombstone on a compacted topic removes one key from that topic, not the copies in other topics or downstream.

Crypto shredding gets there through the keys instead. Gateway derives the key ID from a field in the record, such as the customer ID, so each customer's data is encrypted under its own key. Those keys are stored encrypted by a master key in your KMS, and deleting one makes that customer's records permanently unreadable through Gateway, whatever the topic's retention.

Two things to plan for before you rely on it:

• The identifier has to be in the record when it's produced. Gateway builds the key ID as it encrypts, so a record without the customer ID can't be shredded on its own later.
• Deletion is permanent. There's no undo, so treat the store of per-customer keys as production state, because it decides what remains readable.

What a Proxy Doesn't Fix

A proxy only protects traffic that goes through it. If a client can still reach the brokers directly, encryption on produce still protects what's stored, but masking and access rules on consume don't apply to that client. The network has to send clients through the proxy for those rules to hold.

Once the fraud service has the clear address, what it writes to its own database or logs is that team's responsibility. Crypto shredding makes the Kafka copy unreadable, not the copies a consumer already made.

Masking in Console is for people, not applications. Console data masking policies hide fields from Console users based on their groups, and the underlying Kafka data stays unchanged. Console is the control plane: it decides who sees what in the UI. Gateway is the data plane: it changes what applications receive.

Full-payload encryption catches fields nobody declared, but anything that reads the brokers directly, such as a stream processor, can't process those records.

Where to Start

A short plan that works on most clusters:

1. List the topics that carry personal data and who reads each one. Record the sensitive fields in a list, in schema tags, or both.
2. Mask before production data leaves production. Copies into test environments go through masking on consume.
3. Encrypt fields whose audience is smaller than the topic's. Grant decryption per group and keep everyone else on ciphertext.
4. Add a fallback for undeclared data. Full-payload encryption on unclassified topics keeps a missed field out of the clear.
5. Key by customer where erasure applies. Put the customer ID in the record at produce time, before the first deletion request.

Kafka decides access per topic, and PII needs decisions per field and per reader. Write down who must never read each sensitive field, then pick the mechanism that keeps that reader out: masking for UI users, consumer apps and test copies, encryption for anyone who can reach the stored data, and a key per customer for erasure.

To see field-level encryption and per-group decryption on your own topics, talk to us.

---

Related: Designing Kafka Applications for Data Privacy Compliance → · Crypto Shredding in Kafka → · Stop Building Kafka Encryption Libraries →