Kafka, AI Agents, and Data Quality at LinkedIn
Harshada Yesane is a Senior Software Engineer at LinkedIn, where she works on the open-source Brooklin project and Kafka at scale, and is exploring AI agents to support on-call rotations. She recently gave a talk at Current London on building reliable CDC at trillion-message scale.
In this conversation
- →Working on LinkedIn's open-source Brooklin project and Kafka at scale
- →Building AI agents to take some of the weight off 3am on-call
- →Why the agent is the easy part, and the data underneath is the hard part
- →Data freshness versus data correctness when hundreds of systems depend on you
- →Why she still calls Kafka paramount for any streaming solution
"Building an AI agent is not a difficult thing. The data we have, we have to make sure it's very clean and appropriate, so our agents are more intelligent on top of it."
Interview highlight
Data correctness is the hard part
Building the agent, or the data pipeline that feeds it, was never the hard part. Ensuring that data is correct, in real time, is.