Where Should Your Kafka Encryption Keys Live?

"Aurélie, Vault can't be in our critical path."

I hear some version of that sentence on a lot of the Kafka encryption projects I work on, in English and in French, and it goes the same way in both languages. It usually comes from a security team that has done things properly: the keys sit in a KMS, and nobody copies them around by hand.

Then we start encrypting Kafka, and where the keys live becomes an availability question too. My answer, after a few of these projects, is to put the KMS in the data path on purpose, behind a cache, and to spend the design time on what happens when the cache misses.

Encrypting Kafka means asking a KMS for keys

Kafka encrypts traffic with TLS and can sit on encrypted disks, but it can't encrypt a field before the broker stores it. KIP-317, the proposal for end-to-end encryption, is still under discussion. So field-level encryption happens outside the broker, in client libraries or in a proxy.

The standard design is envelope encryption. The encryption layer encrypts each record with a data key, then asks the KMS to encrypt that data key with a master key that never leaves the KMS. The encrypted data key (the EDEK) travels with the record, so the decrypting side can tell which master key to ask for. HashiCorp sums up its Transit engine, a common choice here:

"The transit secrets engine handles cryptographic functions on data in-transit. Vault doesn't store the data sent to the secrets engine." — HashiCorp Vault documentation

Asking the KMS on every record would make it as busy as your busiest topic, so encryption layers keep data keys in a cache for a while:

So the KMS is in the path only on a cache miss, which is a much smaller dependency than "Vault is in our critical path" makes it sound.

"Load the keys at startup, and Vault can be down"

The alternative I usually hear is a good pattern, and plenty of teams run their applications this way. An operator reads the keys from the KMS and loads them into each pod as environment variables, so pods start and scale even when the KMS is unavailable.

My first question to a team with this setup is how often they rotate their keys. An environment variable doesn't change inside a running pod, so every rotation means a restart, and so does every new topic that needs its own key. For an application that's a rolling deploy. For a Kafka proxy it means every connected client reconnecting, since the proxy holds their long-lived connections. Here are the two designs side by side:

🚫 "If the KMS isn't in the data path, a KMS outage can't hurt us."

It can, just on a different day: the day you rotate a key or onboard a topic. In the meantime, the master key sits in every pod's environment. With a cache, an outage only affects requests that need a key the cache doesn't hold, and you can count those.

A cache makes the KMS a dependency on misses, not on every record

Once the KMS sits behind a cache, what matters is when the cache misses and what each miss does:

Event	Calls the KMS?	What to decide
Produce, data key cached	No	How long a data key stays cached
Produce, new or expired data key	Yes, to encrypt it	What the producer sees if the call fails
Consume, data key not cached	Yes, to decrypt it	Ciphertext or an error for the consumer
Proxy restart or new instance	Yes, for every key in use	How fast a cold proxy warms up

A longer cache rides out a longer KMS outage, and a shorter one makes a rotated or revoked master key take effect sooner. During an outage, the rows that matter are the ones where a call can fail. A well-behaved encryption layer fails the produce and lets the client retry rather than writing plaintext, and what a consumer gets should be a choice you make, not a default you find out about.

Logging in to the KMS and using the keys are two separate steps

A question I get a lot in working sessions: if the encrypting side logs in to Vault with a Kubernetes token, how does the decrypting side get that token? It doesn't need it, because logging in and using keys are two separate steps.

First the encryption layer authenticates to the KMS, and there are several ways to do that. The one I recommend is a Kubernetes service-account token that the operator writes to a file and keeps up to date, so the layer reads a fresh token each time it logs in. A static token also works, but nothing refreshes it. Then, once logged in, the layer asks for keys by name, and the KMS policy decides whether that identity may use them. The decrypting side logs in with its own identity and goes through the same two steps.

In Conduktor Gateway, those are two separate settings in the encryption interceptor:

"kmsConfig": {
  "vault": {
    "uri": "https://vault.internal:8200",
    "type": "KUBERNETES",
    "role": "gateway-encryption"
  }
},
"recordValue": {
  "fields": [
    {
      "fieldName": "email",
      "keySecretId": "vault-kms://vault.internal:8200/transit/keys/customer-email"
    }
  ]
}

Read line by line:

• type: KUBERNETES logs in with the pod's service-account token. Kubernetes writes it to a file and rotates it on its own, and Gateway reads the file each time it logs in, so no restart is needed. A static token in an environment variable only changes with a restart, so I steer people away from it.
• role is the Vault role whose policy decides which keys this login can use.
• keySecretId names the key for this field: one per topic, per field or per tenant, whatever your security team asks for.

The decryption interceptor uses the same KMS settings and can list the fields it decrypts. Scope it to one service account and a support application sees a customer's email while the national ID number stays encrypted. You can be very precise about who sees what.

What Gateway doesn't do for you. It doesn't rotate master keys. You rotate those in your KMS, new data keys use the new version, and older records stay readable as long as the KMS keeps the old one. See the KMS reference.

Replicated data needs its key in the other region

Then there's the conversation that starts with "we're adding a second region". Gateway writes a reference to the key into each record's headers, so a replicated record arrives in the second region still pointing at the first region's key. Whether the proxy there can use it depends on where the key lives:

There are three ways to set this up, and each moves the problem somewhere different:

Setup	What region B does	The catch
One KMS per region	Calls region A's KMS	A cross-region dependency, the thing a second region should remove
One shared KMS	Calls the same KMS as region A	One outage stops decryption and encryption in both regions
The same key in both regions	Decrypts with its own copy	Your KMS has to replicate keys, and the record's reference has to resolve locally

Some KMSs replicate keys for you. AWS calls them multi-Region keys:

"Each set of related multi-Region keys has the same key material and key ID, so you can encrypt data in one AWS Region and decrypt it in a different AWS Region without re-encrypting or making a cross-Region call to AWS KMS." — AWS Key Management Service documentation

The first time a customer asked me about this, my honest answer was that it depends on what your KMS can do with keys, and on whether your security team allows cross-region networking. Knowing both is what lets you design the best solution.

Check what your encryption layer writes into the record, too. An AWS key reference is an ARN that includes the region, so don't assume the replica key gets used on its own. Test it: encrypt a record in region A, replicate it, and decrypt it in region B with region A's KMS switched off. If that works, you're done arguing.

Where to start

You can make most of these decisions before anything is encrypted:

1. List the events that call your KMS, and how often each one happens on your busiest topics.
2. Decide what a consumer gets when a key isn't available. In Gateway, the decryption errorPolicy returns the ciphertext by default (return_encrypted) or fails the fetch (fail_fetch). Pick one on purpose.
3. Log in with a workload identity, not a long-lived token. A rotating token file needs no restart and never sits in an environment variable.
4. Test replication with the source KMS switched off, before the first encrypted topic replicates.

Your keys belong in a KMS your security team runs, and that KMS ends up in Kafka's data path one way or another. Loading keys at startup keeps it out of the runtime path but puts it in front of every rotation and every new topic. I'd rather put it in the path on purpose, behind a cache, and decide up front what a miss does to a producer, a consumer and the other region.

If you're planning encryption on Kafka, speak to me or one of our solutions architects!

---

Related: Bring Your Own Encryption to Managed Kafka → · Crypto Shredding in Kafka →