Kafka's Tricky Parts
Hugh Evans is a Senior Developer Advocate at Aiven, where he looks after the Kafka community and runs the Get Kafka-Nated podcast and newsletter. Before advocacy he was a platform engineer building on Kafka himself. Safe to say he’s shipped it, and now he’s helping the community make sense of it.
In this conversation
- →Why there is no magic number of partitions
- →Sizing partitions to your consumers and producers, not a target number
- →The overhead and confusion that over-partitioning creates
- →The first wall most people hit: authentication and unintuitive errors
- →Understanding Kafka properly before you design a system on top of it
"Troubleshooting Kafka performance issues on hot topics can be tough. You’ve got to know what you’re looking for, but get advice from other people in the community, look at stuff online, and you can pretty quickly start building that toolset for finding the sticking points in your deployment."
Interview highlight
There is no magic number
You need roughly as many partitions as you have consumers and producers. Chasing a magic number just adds overhead and confusion.