# How Do You Handle PII in Kafka?

Say a team adds the customer's email address to the `orders` topic. From then on, every service that reads the topic gets every customer's email, whether it needs it or not.

Only the fraud service needs it. The search indexer and the analytics job don't, and neither does the test environment that gets a copy of production. The cloud provider running your brokers shouldn't be able to read it either.

So handling PII in Kafka starts with one question per field: who must never read it? The answer decides which protection you need.

## Kafka Protects Topics, Not Fields

Kafka ships the controls you'd expect, and all of them are worth turning on. The [security section of the Apache docs](https://kafka.apache.org/documentation/#security) covers authentication, authorization with [ACLs](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns), and TLS for traffic between clients and brokers. None of them looks inside a record.

Here's what each native mechanism does for the `email` field:

| Mechanism | What it protects | What it leaves readable |
|---|---|---|
| TLS | Records on the network | Records on the broker's disk |
| Disk or volume encryption | A disk that leaves the data center | Everything the broker serves to clients |
| ACLs | Which principals can read a topic | Every field of a topic you can read |
| Retention and compaction | Old segments, older values for a key | One customer's records everywhere else |

The ACL row matters most for PII. An ACL grants an operation on a resource, so `READ` on `orders` returns the whole record to the fraud service, the search indexer and the analytics job alike.

Read access on a topic is read access to every field in it.

This works when everyone who reads a topic is allowed to see everything in it. It stops working the day one field has a smaller audience than the topic it lives in.

## You Can't Protect a Field Nobody Has Classified

Before any mechanism, somebody has to know that `email` is in `orders` at all. A question we hear a lot from platform teams is how they'd even find out which topics carry personal data.

There are two common ways to record which fields are sensitive:

- **A list the platform team owns.** It works on any data format, including schemaless JSON, but it goes stale when a producer adds a field and nobody tells the platform team.
- **Tags in the schema.** The producing team marks `email` as PII in its [schema](https://www.conduktor.io/glossary/schema-registry-and-schema-management). It only covers topics with a schema you control, and platform teams tell us they can't count on every producer team to tag its own fields.

Most teams end up with both, plus a fallback for the topics nobody declared at all.

## Decide Who Must Never Read Each Field

With the fields known, go back to `orders` and list the readers who shouldn't see `email`. Each one is stopped by a different mechanism, and each mechanism leaves something open:

| Who must never read the value | Mechanism that stops them | What stays open |
|---|---|---|
| People browsing topics in a Kafka UI | Masking in the UI | Applications and CLI tools |
| A consumer app, or a test copy of production | Masking on consume | The clear value stored on the brokers |
| Broker operators, the cloud provider, backups | Encryption on produce | Consumers you allow to decrypt |
| An analytics team that counts or joins on the field | Tokenization | Places where clear values and tokens meet |
| Everyone, after an erasure request | A key per customer, deleted (crypto shredding) | Copies already decrypted elsewhere |

In the `orders` example, the fraud service is the only reader that needs the clear address. The search indexer only needs masking on consume, the analytics job may need tokens if it counts customers, and the test copy needs masking before it's written. The cloud provider is only kept out by encryption, because masking never changes what the broker stores. Most teams need two or three of these at once, not the one that's easiest to turn on.

## Masking Hides PII. It Doesn't Protect It.

> 🚫 *"We mask PII in Kafka, so it's protected."*

Masking replaces a value with **** as it's read. The record stored in Kafka doesn't change, so masking decides what one reader sees, not what the cluster holds. Masking in a UI is narrower again: it hides the value from people browsing topics and does nothing for applications.

Encryption works the other way round. The value is encrypted on produce, before it reaches the broker, so the stored record holds ciphertext and only the readers you allow can decrypt it:

Masking changes what one reader sees. Encryption changes what the cluster stores.

The difference shows up with a client that connects straight to the brokers. With masking it reads the clear address, and with encryption it reads ciphertext.

Masking still has a clear job: making safe copies. Consume production through masking, produce the result into a test environment, and that copy has nothing to decrypt or reverse.

## Put the Rules Between Clients and Brokers

Masking on consume, encryption on produce and per-reader decryption all need something that sees every record on its way in and out. Kafka brokers store record values as the producer sent them, so that work happens either inside every client or in front of the brokers.

Client-side encryption libraries are where plenty of teams start, but every client in every language has to adopt the [same library](https://www.conduktor.io/blog/stop-building-kafka-encryption-libraries), and one that skips it writes plaintext.

The other place is a Kafka-protocol proxy that every client connects through, so applications keep their code. [Conduktor Gateway](https://www.conduktor.io/gateway) implements this with interceptors for [encryption, masking, tokenization and crypto shredding](https://docs.conduktor.io/guide/use-cases/kafka-encryption). An interceptor can be scoped to a service account, a group or a Virtual Cluster, which is how per-reader access works. Here the `fraud` group can decrypt `email` on `orders`, and clients outside that group keep reading ciphertext:

```yaml
apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: decrypt-email-for-fraud
  scope:
    group: fraud
spec:
  pluginClass: io.conduktor.gateway.interceptor.DecryptPlugin
  priority: 100
  config:
    topic: "orders"
    kmsConnectionName: PROD_VAULT
    recordValueFields:
      - email
```

`scope.group` limits the interceptor to clients in the `fraud` group, and `topic` limits it to `orders`. `recordValueFields` names the one field this group may decrypt. `kmsConnectionName` points at the KMS that holds the master key, which Gateway uses without ever seeing it.

## Erasure Needs a Key per Customer

Kafka can't delete one customer's records on request. Retention removes data by age or size for a whole topic, and a tombstone on a [compacted topic](https://www.conduktor.io/glossary/kafka-log-compaction-explained) removes one key from that topic, not the copies in other topics or downstream.

[Crypto shredding](https://docs.conduktor.io/guide/tutorials/crypto-shredding) gets there through the keys instead. Gateway derives the key ID from a field in the record, such as the customer ID, so each customer's data is encrypted under its own key. Those keys are stored encrypted by a master key in your KMS, and deleting one makes that customer's records permanently unreadable through Gateway, whatever the topic's retention.

Two things to plan for before you rely on it:

- **The identifier has to be in the record when it's produced.** Gateway builds the key ID as it encrypts, so a record without the customer ID can't be shredded on its own later.
- **Deletion is permanent.** There's no undo, so treat the store of per-customer keys as production state, because it decides what remains readable.

## What a Proxy Doesn't Fix

A proxy only protects traffic that goes through it. If a client can still reach the brokers directly, encryption on produce still protects what's stored, but masking and access rules on consume don't apply to that client. The network has to send clients through the proxy for those rules to hold.

Once the fraud service has the clear address, what it writes to its own database or logs is that team's responsibility. Crypto shredding makes the Kafka copy unreadable, not the copies a consumer already made.

> **Masking in Console is for people, not applications.** Console [data masking policies](https://docs.conduktor.io/guide/conduktor-in-production/admin/data-masking) hide fields from Console users based on their groups, and the underlying Kafka data stays unchanged. Console is the control plane: it decides who sees what in the UI. Gateway is the data plane: it changes what applications receive.

Full-payload encryption catches fields nobody declared, but anything that reads the brokers directly, such as a stream processor, can't process those records.

## Where to Start

A short plan that works on most clusters:

1. **List the topics that carry personal data and who reads each one.** Record the sensitive fields in a list, in schema tags, or both.
2. **Mask before production data leaves production.** Copies into test environments go through masking on consume.
3. **Encrypt fields whose audience is smaller than the topic's.** Grant decryption per group and keep everyone else on ciphertext.
4. **Add a fallback for undeclared data.** Full-payload encryption on unclassified topics keeps a missed field out of the clear.
5. **Key by customer where erasure applies.** Put the customer ID in the record at produce time, before the first deletion request.

Kafka decides access per topic, and PII needs decisions per field and per reader. Write down who must never read each sensitive field, then pick the mechanism that keeps that reader out: masking for UI users, consumer apps and test copies, encryption for anyone who can reach the stored data, and a key per customer for erasure.

To see field-level encryption and per-group decryption on your own topics, [talk to us](https://www.conduktor.io/contact/demo?src=blog-how-to-handle-pii-in-kafka).

---

Related: [Designing Kafka Applications for Data Privacy Compliance →](https://www.conduktor.io/blog/designing-kafka-applications-for-data-privacy-compliance) · [Crypto Shredding in Kafka →](https://www.conduktor.io/blog/crypto-shredding-in-kafka-a-cost-effective-way-to-ensure-compliance) · [Stop Building Kafka Encryption Libraries →](https://www.conduktor.io/blog/stop-building-kafka-encryption-libraries)