Optimizing Kafka for Throughput
Adrian Woodhead is a Distinguished Engineer at Human Security, where bot-detection signal data from across the web flows through Kafka to decouple their front-end and back-end systems. He wrote up the throughput experiment in this conversation on the Human Security engineering blog.
In this conversation
- →How Human Security uses Kafka to decouple bot-detection signal data
- →Building a load-test harness to emulate production traffic
- →Why gzip pinned CPU through thread contention under load
- →How switching to the less efficient snappy raised throughput
- →The benchmarking discipline that made the result trustworthy
"Snappy is less efficient than gzip, but it uses the CPU less. And we got much higher throughput."