The Revolution From Current San Francisco 2026
When you drive into SF, you see a ton of billboards, all about AI agents! We were no exception: we launched Conduktor Agents at Current San Francisco.
Platform teams opened with the security of our agents: who controls an agent? What exactly does it do? Where does the data go? The revolution we came back with is ownership: who owns each agent, each failover and each bill.
The other topics were disaster recovery, Kafka proxies and chargeback. More on these below.
What is Streamhouse? Agents need fresh data
The Confluent keynote was about "Streamhouse", a name given to a streaming architecture back in September by Confluent and others At Current, its key message was that an agent is only as good as how fresh its context is, which is the reason data streaming exists. Confluent used a classic example, a flight booking: an agent working from hour-old availability tries to reserve a seat that's already been taken.
Yes, fresh context matters, but an agent that calls an LLM on every event makes no sense: at 10,000 records a second, that's 864 million model calls a day. Latency, quotas and cost break long before the agent helps.
Keep agents off the hot path. Deterministic code and ML models process the stream, and only the few events worth a decision, such as an anomaly to investigate, reach a model. We describe that design in Where AI Agents Belong in a Kafka Architecture.
The missing part was agentic governance: fresh data is great, but what is the agent allowed to do with it? That's what platform teams asked us right away:
Off stage, nobody we talked to mentioned Streamhouse, except to bring up our post about it, which they remembered as... opinionated, let's say.
Agent security came first
Have you seen all the news about security loopholes and CVEs? Security is definitely no joke with super-intelligent agents around.
We launched Conduktor Agents to drive more adoption of streaming in organizations. You shouldn't need an army of Kafka experts to get demonstrable value out of your Kafka.
It was no surprise that platform teams asked us about control and security before even discussing agentic features! Which model, which tools, whose key, where the data goes, what the agent may access and change. In our Voices of the Kafka Community interview with EchoStar, Akshara, a Data Engineer at EchoStar, said the same:
"Scoping the skills of the agent is one of the biggest things I've focused on. Saying, okay, this agent is allowed to do this."
How Conduktor Agents answer those questions
Nothing better than a clear summary in a table:
| Question | How Conduktor Agents work |
|---|---|
| Which model? | Bring your own model. Conduktor runs in your environment. |
| Where does the data go? | Only to the model you connect, on your own account, and nowhere if you host the model yourself. Nothing goes through Conduktor. Agents work mostly from metadata and read messages only when a task needs them. |
| Which identity? | Its creator's. It can never exceed the permissions of the person who created it (app developer or platform team). |
| Which tools? | Only a defined subset of MCP tools per task. |
| What can it change? | Defined per task: report, recommend, request approval, or carry out an explicitly permitted change. |
| How do we roll it out? | Agents start disabled. You run one manually before enabling autonomous execution. |
| How do we audit agents? | Every model turn, decision, and tool call is traced in Conduktor. |
Evals and action logs for Kafka agents
Always give an agent a way out. That's a big lesson. In How to NOT Evaluate an AI Agent, Matt, our VP of Engineering, showed what we found testing agents on real Kafka clusters. A goal written like a recipe pushes the agent to finish it at any cost: agents tested boundaries and invented topics and data to prove they had succeeded. That's literally cheating.
Many people were surprised by this one: let the agent report that nothing needed doing. It's simple, but without it, your agents will cheat or burn tool calls and tokens for no reason. The other lesson was to write goals as outcomes, not as tasks (do this, then that): let the LLM find its way from the context it retrieves. With simple advice like this, our evaluation pass rate went from 35% to 77%, and that was just the beginning.
Then Ron continued with his talk Kafka as the Agent Action Log, where he showed that an LLM asked the same question twice can give a different answer the second time. His example was a crash between processing and commit: Kafka delivers the message again, and the agent asks the model again. The takeaway is to write the agent's decision to a Kafka topic before it acts. After a crash, the agent recovers from that one recorded decision instead of asking the model again.
The TL;DR: agents will only get adopted on Kafka if their behavior is predictable, constrained, and auditable.
How to implement disaster recovery? One application at a time
Disaster recovery is the most fun to discuss. Engineers love this kind of problem: architecture, edge cases, where it breaks, which tradeoff to take. But eventually, every technical choice in it turns into an organizational discussion.
Larger organizations want topic failover or topic migration per application, in stages, instead of moving a whole cluster at once. The usual wish is to start with the most critical applications, because those have the strictest SLAs. As the discussion goes on, we show that a platform team needs three kinds of metadata:
- Lineage. The dependency graph: which topics each application produces to and consumes from, and which applications share them.
- Ownership. The team that owns each application and topic, and signs off on its failover.
- Criticality. Which applications have the strictest SLAs and should move first.
Miss one link and the failover plan will... fail. You can't fail over a producer and leave its consumers behind on the original cluster. Replication is often the easiest part (and the checkbox you show whoever validates the disaster recovery plan, or DRP). Coordinating applications and teams is the real hurdle to making a DRP work in real life.
We usually have this discussion with our customers: a team asks for topic failover and we end up mapping lineage and ownership first. Conduktor Console holds that metadata with its Stream Lineage and federated ownership, and Conduktor Gateway handles client routing between the primary and backup clusters without app reconfiguration or restart. We go further in our Kafka Disaster Recovery Beyond Replication ebook and DR readiness checklist.
Kafka proxies: an accepted pattern
Years ago, we were spending a lot of time explaining what a Kafka proxy does and why it's important. These days, it's no longer a question. Platform teams know the pattern and its tradeoff: one more hop in the data path, in exchange for control. That hop lets them manage risk, protect the cluster, monitor usage, and route or fail over clients, like any HTTP proxy you know.
A proxy also hides what runs behind it: clients connect to the Gateway, whatever cluster or Kafka provider sits behind it. That abstraction is often what teams want. And if you care about the latency impact, we published our benchmark not long ago, and we can confidently say we're improving it significantly as we speak, so it shouldn't be a blocker.
Another benefit of a Kafka proxy is sharing streaming data with partners. This is still a hot topic. Many companies share from their analytical systems instead, and partners lose the near real-time value of the data. Organizations often realize that streaming is the way to share data: data is worth most when it's fresh.
We also heard from teams sharing streams with partners by running MirrorMaker (or an equivalent) and a dedicated cluster for each partner. Each new partner adds:
- A cluster to pay for and maintain, plus a replication process.
- Another copy of the data, with its own ownership, authentication and authorization rules.
This always translates to delays before the data reaches a new partner. A proxy that exposes selected topics, aliases them and masks fields lets partners read from the cluster you already run, in minutes (a great base for partner self-service, if you want our CTO's opinion). Our guide to Kafka data sharing options compares the alternatives.
Speaking of our CTO: Stéphane's talk, Wireshark for Kafka, showed that most Kafka incidents start with misconfigurations or misbehavior on the application side, and walked through the anti-patterns behind 99% of Kafka support tickets:
- An app creating a new producer for every produce request (which led to a Kafka outage).
- Calling
send().get()in a loop. - Idle consumers generating thousands of fetch requests per second.
- A missing ACL that resulted in 100,000+
InitProducerIdrequests in just ten seconds.
The room, full of people who had used Wireshark (so clearly advanced), was surprised by how easily a few bad client configurations can hit the cluster hard. Most teams carry that load without realizing it, until an outage hits and they need to find the root cause. Unfortunately, we see it quite often: prospects regularly come to us right after an incident, when the damage is done.
A Kafka proxy sees every one of those requests and misbehaviors. It gives the platform team visibility and control over all its clients, and one place to fix the whole fleet.
Kafka costs: chargeback per team
FinOps, cost and efficiency are on everyone's mind. With Kafka, the discussion often starts with chargeback (or rather showback) to each team that uses Kafka.
To implement chargeback, you need at least two things:
- Usage metrics per resource (topics, partitions, client traffic).
- An owner for each resource.
Without them, the platform team defends one aggregated bill (the Kafka platform itself), and application teams never see their own slice of it.
Chuck, one of our Solutions Architects, attacked the same bill from the partition side in The Partition Tax. His playbook has four layers:
- Prevent. Block unnecessary partitions with provisioning limits.
- Clean. Remove unused resources on a schedule.
- Optimize. Size partition counts during application design.
- Patch. Fix at the platform level what the other three miss.
Three of the four layers are application-team work. A platform team can set the limits, but application teams still need to see what they've provisioned, whether they use it, and what it costs.
Chargeback usually arrives with a wider IT FinOps initiative, and Kafka fits it poorly. Most organizations treat Kafka as one IT service, a black box with one line on the IT bill, and expect the same accountability as for a VM owned by one team. But a Kafka cluster serves many teams: it's shared infrastructure.
In a tight market, companies cut what costs too much for value that's hard to prove, and streaming value sometimes is (we've already seen companies remove streaming and go back to API calls only, because it's "simpler" and "cheaper"). A platform that shows up as one big cost center is an easy target. Our field guide to hidden Kafka costs goes deeper.
TL;DR: fix your ownership
All the themes above (AI agents, disaster recovery, data sharing and chargeback) come down to an ownership question:
| Topic | What ownership tells you |
|---|---|
| Agents | Which identities have permission to access or change particular resources |
| Disaster recovery | Which applications depend on which topics, and who can authorize a failover |
| Data sharing | Which workloads can see which data |
| Chargeback | Which teams are responsible for the resources being consumed |
If you want to join the proactive Kafka teams that let agents act on their clusters, please reach out.
Related: What We Learned at Current London 2026 → · Kafka AI Agents Should Behave Like Kubernetes Controllers → · Introducing Conduktor Agents →