A pod dies. The ReplicaSet controller notices that the count is off by one, schedules a replacement, and keeps watching. Nobody gets paged, nobody approves the change, and nobody calls it AI.
Now look at the Kafka cluster next to it. A topic is retaining data for 30 days when the platform standard says seven. A connector has been failing and restarting all week. Someone added an email field to a topic that was never classified as sensitive. Those problems are found by a person, if at all, and another person then decides what to do and runs the command.
Kafka's routine work has stayed manual because much of it needs context, not just a rule. That gap is what agents are for, and the model for trusting them is already running in the same cluster.
Your cluster already runs autonomous agents
A Kubernetes controller has a straightforward job. It observes the current state, compares it with the desired state, and acts when the two differ. Then it does it again. It also runs under a defined identity with limited permissions: a ReplicaSet controller can create and remove pods, and it doesn't get permission to read every secret in the cluster just because it needs to reconcile a deployment.
The same pattern shows up all over Kubernetes:
| Controller | Watches | Can do |
|---|---|---|
| ReplicaSet | Pod count vs. desired count | Create and delete pods |
| Horizontal Pod Autoscaler | CPU, memory, custom metrics | Change replica counts |
| cert-manager | Certificate expiry | Issue and rotate certificates |
| Argo CD (auto-sync) / Flux | Git state vs. live state | Apply the difference |
The reconciliation loop isn't unusual. Kafka is mostly missing the governed "act" step.
A lot of Kafka operations are still manual
Kafka teams obviously automate plenty already. Provisioning, deployment and alerting run through CI/CD and GitOps, and operators like Strimzi go further and reconcile declared topics and users the same way a ReplicaSet reconciles pods, which covers provisioning. But there's another class of work that tends to sit around until somebody notices it.
A few examples we hear regularly from platform teams:
- Retention creep. A topic is keeping 30 days of data when the platform template says seven. Nothing is technically broken, so it stays that way until somebody looks at the storage bill.
- Orphaned topics. A project disappears, but its topics don't. Ownership gets lost and cleanup becomes a periodic exercise.
- Connectors stuck in restart loops. A connector fails, restarts, fails again, and never stays down long enough for the usual alert to tell the whole story.
- Sensitive data appearing in the wrong place. An upstream team adds a new field and suddenly personally identifiable information is flowing through a topic that nobody classified for it.
None of these problems is especially difficult for someone who knows Kafka and is looking at it. The problem is getting someone to look. The Kafka platform teams we speak with are small relative to the application teams they support: three or four people for a platform used by fifty or more teams, and they aren't going to inspect every topic, consumer group, connector, and configuration change every morning.
Kafka tooling already automates a little of this work. Connector auto-restart is one example: a connector fails, the system restarts it, and nobody needs to approve the action. That works because the action is easy to define.
The rest of that list is the same kind of work: routine and unwatched. That's where automation should start, not with the incident everyone is already watching.
"But we already have GitOps"
GitOps is useful, but it solves a different problem. It automates applying declared state. What gets harder is deciding what should happen when live state no longer matches that declaration.
🚫 "We have GitOps for Kafka. It's automated."
Maybe someone changed a topic configuration manually by mistake. Maybe they made an emergency production fix and intentionally haven't updated Git yet. A diff can tell you that the states disagree, but it can't tell you why, and that difference matters because blindly reconciling the configuration may undo a deliberate operational change.
Rules are easy. Exceptions are hard.
Traditional controllers work best when the correct answer is deterministic. Desired replicas: three. Observed replicas: two. Create one. Kafka operations often get less tidy.
Take the stale-topic rule in Insights. A topic can be flagged when nothing has written to it for seven days and no active consumer group appears to be using it. That's useful information, and it still doesn't mean the topic should be deleted. Historical data may still be read occasionally, a consumer may store offsets somewhere other than Kafka, and a compacted topic can be quiet for a long time while still containing important state. So the rule can confidently say "this deserves attention." It can't confidently say "delete it."
A language model has almost the opposite problem. In one of our early experiments, we gave a model cluster information and asked for remediation advice. It happily produced a prioritized plan without understanding how the health score it was optimizing had been calculated. Some recommendations were sensible. Others were guesses presented with the same confidence. The difference between a guess and a judgement is context, the schemas, lineage, ownership, ACLs and recent changes, and a model only has the context it's given.
Use rules where the answer is deterministic, use a model where additional context or interpretation is actually useful, and keep permissions separate from both. A rule shouldn't get permission to delete a topic simply because its condition matched, and a model shouldn't get permission because its answer sounded confident. The identity running the automation should determine what it's allowed to do.
Most of this work doesn't need a model at all:
- A scheduled script can find resources with no owner, once ownership is recorded somewhere.
- A diff can detect a topic that drifted from its template.
- A simple check can catch a bad relationship between
min.insync.replicasand replication factor. - A model becomes useful when the rule runs out of context: is this consumer actually stuck, or just slow? Is this topic abandoned, or intentionally kept as a standby?
The implementation can vary. The governance model shouldn't.
A bounded agent can be easier to govern than the status quo
"Do we really want an agent making changes to Kafka?" is a reasonable question. But the comparison shouldn't be between an agent and perfect human governance. It should be between an agent and however the work gets done today.
Platform teams have described environments where engineers use curl, kcat, shared credentials, or broad admin accounts to get things done. One person called it the "wild west." We've also repeatedly heard about the same audit problem: an action is logged to a shared account, so you know which credential performed the operation but not which person was behind it.
Compare that with a workload that has its own identity, access only to the APIs it needs, permissions that can be revoked independently, and an audit record of every call. That isn't less governed than a person using a shared admin credential. In many environments, it's more governed.
The security requirements customers give us for AI access to Kafka tend to be very practical: start read-only, let permissions follow the user or workload, and make individual tools easy to disable.
We recommend starting with read-only access, which our own MCP server allows: it acts with the permissions of the person whose token it uses, so people and assistants can look before anything changes. Write access is the next step, granted one action at a time, and an automation that runs on its own should get an identity of its own when it gets there.
Some teams are already well past the theoretical discussion. We've spoken with one whose AI bot handles much of first-line support before a human sees a ticket, and the next capability they want is access to Kafka topic contents. At that point, the useful question is less "will agents touch Kafka?" and more "under whose identity, with which permissions, and with what record when they do?"
Autonomy should be granted action by action
Nobody needs to give a new automation broad write access on day one, and Kafka automation can follow the same ramp Terraform users already know. Restarting a failed connector isn't equivalent to deleting a production topic, and pausing a development consumer group isn't equivalent to resetting production offsets. So instead of asking whether an agent is autonomous, ask whether a particular action should be.
You can roughly evaluate an action along two dimensions: how much can it affect, and how easy is it to undo? A failed connector restart is contained and reversible. Throttling a noisy producer affects more traffic but can still be reversed. Lowering retention on a production topic can permanently remove data, and deleting a topic is more consequential again. Those actions shouldn't all have the same approval path.
Autonomy belongs to the action, not the agent. Everywhere on that chart the automation finds and proposes, with its reasoning attached. What changes is who executes, and the team decides where the lines sit.
Start with the boring, reversible operations. As trust grows, individual actions move from "suggest" to "act and notify" to fully automatic. Nobody gives a new engineer permission to delete production volumes on their first morning, and Kafka automation doesn't need a different standard.
Kubernetes automated routine operations early because its decisions could be written as deterministic rules, and Kafka's couldn't. Models now give us a tool for those ambiguous cases, but the model isn't the important part. Point it first at the work nobody is watching, and put any automation, whether it's a script, rule, model, or controller, behind the same things we already know how to trust: an identity, a narrow permission set, and a complete audit trail. That is what a Kubernetes controller has had all along.
Related: Kafka Is the Last Infrastructure Without a Hypervisor → · AI for Kafka Operations → · Conduktor MCP + Skills for AI Agents →