Benchmarking the Conduktor Gateway: what a Kafka proxy really costs
A month ago we wrote about how we test the Conduktor Gateway: millions of protocol permutations, test suites driven by real Kafka clients, byte-precision assertions. That was functional testing. It proved the proxy does the right thing. It said nothing about what that correctness costs you. Today, we talk performance.
A Kafka proxy is a reverse proxy
Like any reverse proxy (NGINX, Traefik, Kong, Envoy), a Kafka proxy adds a hop to your architecture, and you pay for it.
It is the same model you already use for HTTP. A passthrough proxy forwards traffic and costs almost nothing on throughput; you pay real CPU only when you make it read and rewrite what flows through. A Kafka proxy is the same machine, run the same way: several gateways, stateless, so clients reconnect to the survivors if a pod goes, just like your Kafka brokers.
One difference: even in "passthrough" mode, a Kafka proxy is not just dumb byte-forwarding at L4, it works at L7. It parses Kafka requests and can rewrite metadata such as advertised listeners. And it does more than an HTTP request/response proxy: it terminates TLS on both legs (client to Gateway, Gateway to broker) and tracks long-lived, stateful produce and fetch sessions and consumer-group coordination. Once you start altering the payload, as encryption does, it gets more expensive, the same way an HTTP body-rewriting or content-inspecting proxy costs more than a plain hop. Everything below is that model, measured.
The short version
What to consider when you add a Kafka proxy: throughput, tail latency, and CPU.
The results below run over TLS, not plaintext, which flatters a benchmark but is useless because it is not production-representative. Without payload encryption, the proxy adds no throughput cost over raw Kafka: at every workload it sustained the same offered load as the direct-to-Kafka baseline. So throughput is not where the cost shows up. Latency and CPU are.
End-to-end p99 (the tail latency) rises a little, from the extra hop in the path, and only +3% once it is spread across 3 gateways with 50 topics:
Finally, the peak CPU a single gateway pod draws under each workload, the real cost of the new component:
Who runs Kafka in PLAINTEXT?
Nobody. Not in production. And yet plenty of the Kafka benchmarks you will read run over plaintext, OpenMessagingBenchmark's own default workloads among them: no authentication, nothing encrypted on the wire. It only flatters the numbers.
Why? Over plaintext, a Kafka broker serves reads with sendfile() zero-copy, copying bytes straight from the page cache to the socket without passing them through user space. In production, with TLS, that path is gone: every byte has to be pulled up and encrypted in user space. Plaintext benchmarks measure a code path a real cluster never uses.
We are not chasing a headline number. To stay close to production, every connection in these runs is SASL + TLS, end to end: client to Gateway, and Gateway to Kafka.
🚫 A plaintext, single-partition, no-durability benchmark posts an enormous number and measures a cluster nobody runs. Impressive, and irrelevant.
What a benchmark means to us
A benchmark is not a leaderboard score; it depends on too many conditions and use cases. It is only a measurement. Here are our principles:
- Secured. SASL and TLS, because that is how Kafka runs in production.
- Neutral settings. A default-config Kafka cluster and the standard OpenMessagingBenchmark (OMB) client settings, identical on both sides. We measure the cost of the Gateway, not settings picked to make it look good.
- Tails, not averages. We read the result off p99 (the slow tail), not the mean. Averages are useless here: they hide the real traffic shape and its worst corners.
- Watch the baseline, not just the proxy. If a run's direct-to-Kafka baseline is itself saturated, the gateway-vs-baseline comparison is meaningless, so we gate on baseline health and discard those runs.
- Like-for-like. Every configuration runs on the same hardware at the same offered load, a fixed target rate below saturation.
- Report the cost. Passthrough and encryption behave nothing alike, so we measure them separately.
You can also read how we test the proxy for correctness, a magnificent piece of engineering in its own right.
Our setup
Everything runs on AKS.
- Each node is a single
Standard_D8ds_v6(8 vCPU, 32 GiB RAM, NVMe plus premium SSD). - Each latency-critical component gets its own VM, to avoid noisy-neighbour contention.
- Gateway pods are capped at 2 vCPU / 4 GiB.
- Brokers run Confluent with RF=3, on 500 Gi of premium SSD each.
- Messages are 1 KiB (opaque bytes), runs are a 2-minute ramp followed by a 5-minute sustained window, repeated 5 times.
It uses OpenMessagingBenchmark (OMB) with production-realistic client settings, applied identically to the baseline and the proxy:
| component | version / setting |
|---|---|
| Conduktor Gateway | 3.19.1 |
| Client settings (OMB) | acks=all, linger.ms=1, batch.size=128 KiB |
| Durability | RF=3, min.insync.replicas=2 |
| Security | SASL + TLS (SASL_SSL) |
| Encryption | full-payload AES envelope encryption |
| Payload | 1 KiB |
- baseline clients talk to Kafka directly, no Gateway in the path.
- plain-proxy traffic passes through the Gateway, payload untouched.
- encrypt+decrypt the Gateway encrypts payloads on produce and decrypts on consume.
We care about these 2 metrics:
- produce is send-to-ack: how long a producer waits for its write to be acknowledged.
- e2e is end-to-end: from a message being sent to a consumer reading it back.
Both are reported at several percentiles: p50, p95, and p99 (at p99, 99 requests out of 100 are at least this fast).
The passthrough cost
We run four workloads, from a single hot topic up to a realistic multi-gateway fan-out.
The goal is to measure the passthrough (plain-proxy) cost against the baseline. The fourth, encryption, is heavier and gets its own section at the end.
Workload 1: One Hot Topic
Say you are sharing one heavy topic with a partner. All the load funnels through a single topic: one producer, one consumer, one Gateway, three brokers.
| metric | baseline | plain-proxy | Δ |
|---|---|---|---|
| Produce latency | |||
| p50 | 1.2ms | 1.5ms | +0.3ms |
| p95 | 4.1ms | 7.6ms | +3.5ms |
| p99 | 21.3ms | 28.7ms | +7.4ms |
| End-to-end latency | |||
| p50 | 21.0ms | 20.0ms | -1.0ms |
| p95 | 35.0ms | 35.0ms | 0ms |
| p99 | 45.0ms | 52.0ms | +7.0ms |
| Throughput & resources | |||
| throughput | 120k msg/s (123.1 MB/s) | 120k msg/s (123.1 MB/s) | -47 msg/s |
| peak CPU | - | 0.46 cores | - |
| peak MEM | - | 0.47 GiB | - |
Workload 2: More Topics, More Producers, More Consumers
We increase the traffic now: ten topics, ten producers, ten consumers, still one Gateway. The idea is to stress the Gateway's connection concurrency, simulating constant Kafka traffic across many applications.
| metric | baseline | plain-proxy | Δ |
|---|---|---|---|
| Produce latency | |||
| p50 | 2.1ms | 2.6ms | +0.5ms |
| p95 | 28.5ms | 37.9ms | +9.4ms |
| p99 | 192.5ms | 211.6ms | +19.1ms |
| End-to-end latency | |||
| p50 | 54.0ms | 60.0ms | +6.0ms |
| p95 | 107.0ms | 118.0ms | +11.0ms |
| p99 | 234.0ms | 304.0ms | +70.0ms |
| Throughput & resources | |||
| throughput | 260k msg/s (266.8 MB/s) | 260k msg/s (266.8 MB/s) | +64 msg/s |
| peak CPU | - | 1.54 cores | - |
| peak MEM | - | 0.57 GiB | - |
produce p99 of 192 ms, measured with no Gateway in the path, is already high: this is the busiest single-gateway workload, and the brokers under load, not the proxy, drive that tail. The proxy adds only about 10% on top. When tail latency looks bad, the brokers are usually the cause; the proxy is just a passenger. Notice the Gateway CPU: 0.46 cores with one client in the previous One Hot Topic test, 1.54 with ten topics here. A proxy carries a fixed cost per pod (the JVM, TLS state, its event loop) that dominates at low load. Size it as a fixed floor plus a small slope, not a flat cost per client.
Workload 3: Let's go to production
Now we go to production scale: 50 topics × 4 partitions, 50 producers, 100 consumers, served by three gateways in front of five Kafka brokers.
| metric | baseline | plain-proxy | Δ |
|---|---|---|---|
| Produce latency | |||
| p50 | 3.1ms | 3.7ms | +0.6ms |
| p95 | 83.2ms | 90.7ms | +7.5ms |
| p99 | 104.5ms | 128.0ms | +23.5ms |
| End-to-end latency | |||
| p50 | 265.5ms | 267.0ms | +1.5ms |
| p95 | 485.5ms | 490.0ms | +4.5ms |
| p99 | 524.0ms | 540.0ms | +16.0ms |
| Throughput & resources | |||
| throughput | 230k msg/s (235.8 MB/s) | 230k msg/s (236.1 MB/s) | +371 msg/s |
| peak CPU | - | 0.78 cores per pod (2.33 total, 3 pods) | - |
| peak MEM | - | 1.61 GiB | - |
Remember the note on Gateway CPU? Here it is spread across three pods, about 0.78 each, carrying 150 client connections. Each pod does far more work for less CPU than in the previous test: the proxy gets cheaper per connection the bigger it gets, because that fixed per-pod cost amortizes. Scale out, not up.
The latency the proxy adds is small. The wait is dominated by the consumer's fetch cadence and the broker's queue, not by the produce path, so the Gateway's bounded hop at each end, a couple of milliseconds, barely moves a number that large: e2e p99 goes from 524 to 540 ms.
The cost of payload encryption
Encryption is a different kind of work. Not TLS this time, but payload encryption: the Gateway encrypts on produce and decrypts on consume, touching the bytes of every single message.
Why bother, when you already run TLS? TLS and at-rest encryption cover the wire and the disk. Neither touches the Kafka API itself: a single stolen credential authenticates and reads every topic it is allowed, in plaintext, because the broker terminates TLS and serves plain bytes. Payload encryption closes that gap: the data stays ciphertext on the broker, in backups, in logs, and to whoever operates the cluster, with keys the cluster never holds. For regulated teams (finance, healthcare, public sector), that is often what lets sensitive data go on Kafka at all, or move to a managed or cloud cluster they do not fully control.
We test it in isolation: we reuse the same one-hot-topic setup but we add encryption, which is CPU-heavy. To compare latency like-for-like at equal load, both the baseline and the encrypted path produce at a matched 60k target.
For these runs the key-encryption-key lived in the Gateway's in-memory key store rather than an external KMS, so the numbers isolate the per-message cryptographic cost. A production KMS (Vault, AWS KMS, an HSM) adds a key fetch, but the Gateway caches it, so it only happens at startup and on key rotation, not per message. The cost lives in the per-message crypto, not the key fetch.
| metric | baseline (60k) | encrypt+decrypt (60k) | Δ |
|---|---|---|---|
| Produce latency | |||
| p95 | 2.3ms | 244.6ms | +242.3ms |
| p99 | 19.4ms | 505.1ms | +485.7ms |
| End-to-end latency | |||
| p95 | 52.0ms | 469.0ms | +417.0ms |
| p99 | 56.0ms | 748.0ms | +692.0ms |
| Throughput & resources | |||
| throughput | 60k msg/s (61.5 MB/s) | 60k msg/s (61.5 MB/s) | - |
| peak CPU | - | 1.27 cores | - |
| peak MEM | - | 0.76 GiB | - |
The cost is per-message: the Gateway encrypts on produce and decrypts on consume, so latency tracks the crypto work on the path. At 60k it peaked at only 1.27 of its 2 cores, so it was not CPU-bound in aggregate: the cost lands on latency, not on a saturated box. Why the ceiling sits near 74k rather than higher, and how much of the per-message work can be parallelized, is something we are still breaking down; we will follow up with the per-core detail.
The cost of encryption is latency, the price of doing cryptographic work on every message. So turn it on for the topics that carry sensitive data, not the whole cluster. Note that these runs encrypt the whole payload; field-level encryption, which parses the schema to encrypt only chosen fields, has a different profile we did not measure here.
When do you put one in the path?
- For cross-VPC connectivity, routing, and networking, a passthrough Kafka proxy is great, seamless, and cheap, and you get everything else the Gateway can do in return.
- For encryption, budget for it. Scope it to the data that actually needs protecting, because it adds real latency on both produce and end-to-end. It is a trade-off: latency versus central data protection. If you are not latency-sensitive and can absorb an end-to-end p99 near 750 ms under encryption, it is an easy call.
A rough capacity guide: size against the production-scale figure, about 100k msg/s per core (Workload 3). A single lightly-loaded pod does far more, up to ~260k per core on one hot topic.
The interesting part is not really the performance. It is that once a proxy sits in your data path, you can do far more with your Kafka traffic than move it.
If you want the correctness half of this story, it is in how we test the Conduktor Gateway. If you are weighing the architecture, what a Kafka proxy can actually do is the place to start.