Bring Your Own Encryption to Managed Kafka

Every managed Kafka provider encrypts your data at rest (disks, buckets), and every one of them also holds the keys.

This means the provider can technically read your plaintext, because its systems decrypt data to process it. You can't prove independent key control, because you don't control the keys. And because the data is sealed with the provider's own keys, your encryption is tied to that one provider.

There's another model: encrypt the data before it ever reaches Kafka. The broker stores ciphertext, replicates ciphertext, serves ciphertext. The keys live in a system you own. When that's true, your Kafka provider stops being a security boundary. It still runs the cluster; it just no longer holds anything it could read.

Conduktor Gateway does this at the proxy layer. This post covers how it works and the benefits of having a provider that can't read the data you want to protect.

The provider holds your keys

Many teams are happy with just at-rest encryption. But what does it actually defend against?

It protects against one thing: someone walking out of a data center with a physical disk. Good luck stealing one from an AWS us-east-1 data center. (For the full transit / at-rest / end-to-end picture, see Kafka encryption: transit, rest, end-to-end.)

🚫 "Our data is encrypted at rest, so the provider can't read it."

Absolutely not. The provider manages those keys. Their systems decrypt your data to serve it, and an employee with sufficient access can read your data. At-rest encryption stops a stolen drive; it does nothing about the provider itself.

You don't have a security posture if you don't have the encryption keys.

The only way to be serious about it is that:

• data arrives already encrypted
• you hold the decryption keys
• key management is under your control

Your provider handles replication, availability, and throughput. You handle data security.

Encrypt before Kafka sees it

Conduktor Gateway is a Kafka proxy. Kafka apps and frameworks connect to it as if it were Kafka; Gateway connects to the real cluster, and encryption happens in the middle:

• A producer sends normal JSON/Avro/Protobuf to Gateway.
• Gateway encrypts and forwards ciphertext to the broker.
• On the way back, the broker returns ciphertext, Gateway decrypts, and the consumer gets back readable data.

Kafka only sees byte arrays it can't interpret; that's how it's built and what it excels at.

The bonus: you don't need any application code changes either, since Gateway speaks the Kafka protocol.

Gateway is now in your trust boundary. It handles clear data during processing, caches decrypted keys in memory, and holds your KMS credentials. You've moved the sensitive point from the provider to a component you own and run, which is the entire point. Now you have a compliance story, and can even move to a managed service without putting that data at risk:

"How the data moves through Conduktor and up to the cloud, where it's encrypted and what it's not, all documented, and they saw absolutely no problem with that." — a platform lead in healthcare mentioning their auditor

The keys themselves stay external. Gateway never persists them; it integrates with whatever key manager you already run:

KMS provider	Common auth methods
HashiCorp Vault	Token, AppRole, Kubernetes, LDAP, JWT, cloud IAM
AWS KMS	IAM credentials, STS session, default credential chain
Azure Key Vault	Client secret, or the default Azure credential chain (managed identity, env, CLI)
GCP Cloud KMS	Service account, application default credentials
Fortanix DSM	API key, username/password

Envelope encryption

Gateway uses envelope encryption. The point is to keep your master key out of the data path while still encrypting every record cheaply:

On produce:

• Gateway generates a random Data Encryption Key (DEK)
• Encrypts the record with it
• Asks the KMS to wrap that DEK with your Master Key (the KEK, or key-encryption key), producing an Encrypted DEK (EDEK).
• Gateway stores the ciphertext, the EDEK, and a few metadata in Kafka.
• On fetch, it reads the EDEK, asks the KMS to unwrap it back into the DEK, and decrypts.

To avoid a KMS round-trip on every message, Gateway caches DEKs in memory. When rotating the master key in the KMS, Gateway will use it for new records, and existing records stay readable as long as their key version still exists in the KMS.

The cryptography runs on Tink, an open-source library of misuse-resistant cryptographic APIs maintained by cryptographers and security engineers at Google.

Field-level encryption

You may want to encrypt only a few fields and not the whole payload. A customer event might carry an SSN and a card number that must be protected, alongside a country code and a timestamp you want to keep queryable for partitioning and filtering.

recordValue:
  fields:
    - fieldName: "user.ssn"
      keySecretId: "vault-kms://vault:8200/transit/keys/pii-key"
    - fieldName: "payment.card_number"
      keySecretId: "vault-kms://vault:8200/transit/keys/pci-key"

This works with JSON, Avro, and Protobuf.

Key IDs also support templating: resolve the key from a header, a key field, or a value field, and you get per-tenant or per-user encryption, handy for crypto shredding in Kafka:

keySecretId: "vault-kms://vault:8200/transit/keys/tenant-{{record.header.tenant_id}}-key"

What this does: messages with a header tenant_id: acme encrypt under the KMS key tenant-acme-key. Delete this key in Vault and every "acme" message becomes unreadable, without touching the Kafka log, which is immutable anyway. This is the cleanest answer to a GDPR erasure request when using Kafka.

Tips: Don't encrypt the record key on a compacted topic. Compaction deduplicates by key, and encrypted keys may be unique even for the same business key, so compaction won't work. Encrypt the value, or route on a header instead.

Moving providers without re-encrypting

Even if you have no plans to switch providers, holding your own keys has a massive advantage: no one except your own applications can read your encrypted fields, migration or not. It's about ownership and where the trust boundary sits:

Without proxy-layer encryption, your data is sealed with the provider's keys. The encryption is theirs, not yours. Untangling it later means reading everything out, letting them decrypt it with their keys, and re-encrypting it somewhere else.

With Gateway encryption, the data is already ciphertext in Kafka, and the keys are in your Vault or KMS. To move, you only have to copy the ciphertext as-is (with MirrorMaker, for instance) and point Gateway's bootstrap.servers at the new cluster.

Gateway vs client-side encryption

Without a proxy, you can encrypt on the client side, but let's be honest, it comes with a lot of drawbacks. We wrote about it here: Stop building Kafka encryption libraries.

You can add your own library in your applications, or modify serializers like Confluent's CSFLE does. This path is harder than it looks. You're now shipping crypto inside every app, which means:

• code changes in every application
• support for every language and framework you run
• library versioning across all of them
• CVEs to patch, everywhere, before someone exploits them
• sane crypto defaults that each team has to get right
• governance and data quality on which fields to encrypt, plus the metadata each record needs to be decrypted later
• KMS credentials handed to every service that touches the data

And it only takes one app that skips it to write plaintext into a topic that's supposed to be encrypted.

	Gateway (proxy)	Client-side (CSFLE)
Code changes	None; clients connect as-is	SDK integration in every app
Client languages	Any Kafka client	The supported SDKs only
Kafka provider	Any: Confluent, MSK, Aiven, self-hosted	Confluent only (Cloud or Platform)
Where the encrypted DEKs live	In the Kafka record value, next to the ciphertext	Confluent-managed registry (encrypted DEKs only; your KEK stays in your KMS)
Enforcement	Mandatory at the proxy, on the wire	Per client: a producer that skips the serializer writes plaintext
What you run	A proxy in the data path	No extra component (a client library)

Gateway wins when:

• you run mixed or multiple providers
• you can't modify legacy applications
• you want encryption enforced in one place instead of trusted to each client

Client-side has one genuinely good case: when you need managed Flink or ksqlDB to process the data in-cluster. Anything running behind Gateway sees ciphertext for the fields you encrypted, so in-cluster processors can only work on the fields you left in the clear.

Performance and cost

Encryption isn't free, but with DEK cache hits the cipher is cheap. Gateway uses AEAD algorithms with AES-NI hardware acceleration. A few milliseconds of latency is expected on a cache miss, since it needs a round-trip to your KMS (10ms+, depending on the KMS and where it runs).

On top of the crypto, every produce and fetch takes one extra network hop, exactly like adding an NGINX, Envoy, Traefik, or Kong in front of a service, so it's a well-understood cost. Gateway is stateless, so it's simple to scale horizontally.

Summary

1. Encrypt at the proxy, not the broker. Your provider stores ciphertext and never holds the keys, so it can't read what you've encrypted or tie that encryption to itself.
2. Hold your own keys. External KMS integration keeps master keys in systems you control; envelope encryption keeps them out of the hot path.
3. Encrypt fields, not records. Protect ssn and card_number while country and timestamp stay queryable, across Avro, Protobuf, and JSON.
4. Shred by deleting keys. Per-tenant and per-user key templating turns a GDPR erasure request into a single key deletion: no log rewrite, effective as the cached copies of that key expire.
5. No lock-in on the encryption layer. The keys and the ciphertext are yours, so your encryption isn't tied to any single provider.

To go deeper on the parts around this: what a Kafka proxy can do sets encryption next to the governance, audit, and isolation controls that live at the same layer; and if you ever share encrypted topics outside your own walls, sharing data with partners is the best pattern.

Book a demo if you're the one who has to answer for where your regulated data goes, and "the provider encrypts it" isn't an answer you can put your name to. We'll wire field-level encryption to your own KMS, on your data, so you can watch the broker store nothing but ciphertext while the keys stay in your hands.