Easier Said Than Done in Kafka
AZ
Guest
Abubakar Zahid
Backend Engineer, Trayport
Abubakar Zahid is a backend engineer at Trayport, an energy-trading platform, where he has worked on the data streaming and analytics platform since 2023. Running Kafka at scale in a regulated industry means the hard problems find him, whether he goes looking for them or not.
In this conversation
- →Running Kafka at scale on an energy-trading platform
- →Why data lineage is easier said than done in a regulated environment
- →Auditing for missing data, and why it’s a hard question to solve
- →The day-one lesson: exactly-once holds inside Kafka, and gets very hard once data leaves
"It is easier to guarantee exactly-once within the Kafka ecosystem. But once your data leaves that, it is very challenging. I wish I knew that before I started."
Interview highlight
Easier said than done
The two problems that follow Abubakar around: proving where your data came from in a regulated environment, and holding exactly-once once data leaves Kafka.