Kafka Teams Spend Too Much Time on Routine Work

A platform engineer might start the day planning a Kafka upgrade and spend most of it on something else. A connector fails, a team has waited a week for access to a topic, and nobody knows who owns it.

Each request is reasonable, and none is large on its own. But together, they take time away from the work the platform team had planned:

That's a familiar part of running Kafka. As more teams build applications on it, there's more to handle, more to manage, and more to keep track of.

The platform team has to support that growth and keep improving the platform at the same time. The work comes in three forms:

• Work that comes to the team. It arrives through tickets, alerts, and Slack messages.
• Work the team has to go looking for. That includes service accounts with access they don't need, topics without owners, and personal data nobody classified.
• Developers' requests and questions. Requests for access, topics, and schema changes wait for the platform team to approve them. Questions about a stopped consumer or a missing owner wait for whoever knows Kafka.

These are different problems, but they often depend on the same few engineers who know Kafka well enough to handle them. The more Kafka grows, the harder it gets for those engineers to keep up.

Not all Kafka work triggers an alert

When a consumer stops processing messages or a connector keeps failing, there's a clear reason to investigate. An alert fires, an application misbehaves, or a developer asks for help, and the team works out what happened.

But a lot of Kafka upkeep doesn't start with an incident. It's about making sure the platform is still set up, owned, and used the way the organization expects.

A service account might have access to dozens of topics it no longer needs. A topic might lose its owner when a team reorganizes. A producer might add a field with personal data, or a team might keep paying for topics nobody uses. Kafka keeps working normally the whole time:

The information usually exists, but someone still has to look for it, bring it together, and decide whether anything needs to change. It falls into four areas, each with someone who needs to know:

Area	Who needs to know	What to look for
Access and identities	Security teams	Accounts with more access than they use, service accounts nobody uses, and certificates about to expire
Sensitive data	Governance teams	Personal data in topics that shouldn't have it, fields that need masking, and who can read sensitive topics
Usage and cost	Platform and finance teams	Cost by team, why a team's usage jumped, and topics with no traffic
Ownership and health	Platform engineers	Topics with no owner, failed connectors, and schema changes that will break a consumer

The trouble is that these checks don't stay complete for long. An access review might confirm that permissions are right, and the next day someone creates a new service account. A team might record the owner of every topic, and the records go out of date as applications change.

The same goes for sensitive data, usage, and configuration. A producer adds a field, another application starts reading a topic, or a team's usage jumps after a launch. Each change can be worth a look, even when nothing has broken:

Reviewing the platform more often catches these changes sooner, but it means repeating the same investigations more often. And as the number of applications, identities, and topics grows, there's more to review each time.

Better tools help, but much of the work is still manual

Platform teams already have ways to manage this. Monitoring catches failures, self-service makes developers less dependent on platform engineers, and runbooks document how to handle common situations. These tools have made Kafka much easier to run, but they don't remove the work around them.

🚫 "We have monitoring. If something were wrong, we'd know."

Monitoring is what you want when a broker fails or a consumer falls behind. But many security, governance, and cost problems don't send that kind of signal. A topic without an owner keeps serving traffic, and an account with too much access may never cause a visible problem.

Self-service solves a different part of the problem. Developers can inspect topics, check consumer activity, and fix common issues without asking a platform engineer each time. That cuts support requests and lets more teams build on Kafka.

But developers still get stuck when they don't know where to look or what they're seeing means. They might need to know why a consumer keeps rebalancing, whether a schema change is safe, or who approves access to a topic.

Having the tools doesn't mean having the Kafka knowledge to answer those questions. So developers wait for whoever has it, on the platform team or their own, and a new engineer waits longest:

Runbooks document the steps for recurring work, so engineers have a process to follow instead of relying on someone's memory:

runbooks/
  connector-failed.md        check the task error, then the target system
  consumer-lag-climbing.md   check rebalances, then the slowest partition
  access-review.md           compare each account's access with what it uses
  certificate-expiring.md    renew it, then update the keystores

But a runbook still needs someone to recognize when it applies, gather the information, and work through it. Even when the steps are clear, someone has to start the work, and they may need enough Kafka experience to understand the results.

Monitoring, self-service, and runbooks all reduce the effort of running Kafka. What's left is deciding what needs attention, starting the investigation, and seeing it through. Most of that still falls on people.

Not every investigation needs an expert from start to finish

Take a service account that can read 48 topics but has read from only three in the past month. Its access might be too broad, or there might be a good reason to keep it.

A platform engineer would gather its permissions, check its recent activity, find the applications using it, and track down its owner. With that information, the owner can decide whether to tighten the access or leave it:

The final decision needs context that Kafka doesn't have. The account could be for emergency access, or it could support a job that runs once a quarter. Removing access without knowing that could cause a different problem.

But most of the investigation before that decision is repeatable. The same information has to be gathered for other accounts, and gathered again as permissions and usage change. The same split shows up across Kafka operations:

• A sudden jump in usage might be expected after a launch, but someone still has to spot it and explain it.
• A schema change might already be agreed with downstream teams, but working out its impact still means checking who depends on the topic.
• A topic without an owner might not be urgent, but finding it and working out who should own it still takes time.

In each case, gathering the evidence is different from making the decision. The investigation can usually follow the same steps each time, while the decision depends on knowing the application, the business, and the risks.

Today, platform engineers usually do both. They find the issue, collect the information, and work out what happened before they can decide whether to act. And because the platform keeps changing, they repeat many of those investigations.

Giving Kafka experts more time to build

The cost of this work goes beyond the platform team's calendar. An afternoon on recurring investigations is an afternoon not spent improving the platform, onboarding new teams, or making Kafka easier to use.

Developers feel it too. A team launching a new application might be waiting for access to a topic, or for someone to explain a consumer problem. If the people who can approve or answer are busy, the team waits.

Hiring more engineers, improving documentation, and expanding self-service all help. But as Kafka adoption grows, so does the work around it. Adding people doesn't change how much of that work waits for someone to start it. That work could be handled differently:

• Routine checks could run on a schedule instead of waiting for the next review.
• Common investigations could start on their own when something changes.
• The evidence could be gathered before an engineer gets involved.
• Developers could get answers to common Kafka questions, and access requests could arrive already checked.

None of this removes the need for people who understand Kafka. Engineers would still own the architecture, the unusual failures, and the decisions where application and business context matter. They just wouldn't have to do every repetitive step that leads up to those decisions.

As more teams build on Kafka, the routine work grows with them. Less of it should wait for a person to start it. Then Kafka expertise can go to the hard problems and to the new things the business wants to build.

---

Related: Kafka AI Agents Should Behave Like Kubernetes Controllers → · Good Kafka Is Boring Kafka → · Why Kafka Platform Health Requires More Than Monitoring →