Investigating a Leaked Kafka Credential with Conduktor Agents

Rotating a leaked Kafka password is quick, but the new password keeps every grant the old one had. Finding out what those grants exposed, and narrowing them, takes longer and usually more than one team.

Take a payments team with a reporting job on Kafka. Maya, debugging a failed build, posts in the team channel: "set -x was on, so the payments-reporting password is in it. The log is two days old."

Bob, who looks after the reporting job, replies: "Rotating it now. Then I'll check what that account could read." He can change what his team owns, but only sees his team's topics. Priya, on the platform team, can see the whole cluster, and this leak needs both of them.

Bob Narrows the Grant Himself

With the password rotated, Bob opens Claude Code in the team's repo and types "what can payments-reporting read and who owns it". His assistant reaches Console through the Conduktor MCP server and the Conduktor CLI, both using Bob's personal access token, so it sees and changes only what Bob can.

The account belongs to payments-local, the team's application, and it has Read on every topic starting with pay. The team only owns payments-. His assistant spells out what that prefixed ACL means: "a topic created later with a name like payroll or paypal-events would be readable straight away." Bob can't see whether topics like that already exist.

Every film here is live HTML. Click one to pause it and scroll to read every line, or step through it with the bar underneath.

Bob's assistant reads the account, dry-runs it as it stands, then narrows it.

Bob has three ways to go from here:

• Narrow it now. Change pay to payments- and apply. It's quick, but if something depends on the wider grant, it breaks on his watch.
• Hand it to the platform team. Open a ticket and wait. Nothing breaks, but the grant stays wide until someone picks it up.
• Check where the grant came from first. If his team asked for pay on purpose, something of theirs may rely on it.

He goes with the third, and tells his assistant why: "Before I change anything, I want to know whether this grant came from us." His team manages its access through Console's federated ownership, where an application can only grant access to resources it owns. So the assistant writes the account out as it stands and dry-runs it:

$ conduktor apply -f payments-reporting.yaml --dry-run
Could not apply resource ServiceAccount/payments-reporting: At spec.authorization.acls[0]: appInstance `payments-local` doesn't have permissions or ownership to grant Read on TOPIC/pay

Console refuses. payments-local owns only payments-, so nobody set pay through the team's application, and Bob's token can't read the audit log to see who did. The reporting job only reads the three payments topics, so he has the assistant change pay to payments-, dry-run it again, and apply it. He didn't need a ticket or anyone else's time.

Priya Traces the Exposure

Bob still doesn't know what the old grant could reach, so he posts in the platform channel: "What did pay cover? I can only see our topics."

The leak in the team channel, then Bob's handoff to the platform team.

Priya runs the Kafka platform, and she knows security will start with one question: what that password could read for two days. She opens Ask Conduktor on the cluster page, asks what payments-reporting could read before Bob's change and what's in those topics, and adds: "Go by the schemas; don't read any records."

Ask finds Bob's change in Console's audit log, expands the old pay prefix into today's topics, and reads their schemas. Five topics match. Two are payroll topics, and payroll-employee has fields named ssn and hourly_rate. The other three have no schema, and Ask stops there: "I can't tell what they contain without reading records. You asked me not to, so I didn't."

Ask finds the old grant in the audit log, then reads what it could reach.

Priya could hold her answer until she knows whether anyone used the password. That proof is in the broker's authorizer log, if it was set to record allowed requests, or in Gateway's audit log if clients connect through Gateway, and it may not exist at all. So she tells security what she knows now: for two days, the leaked password could read sensitive payroll data, including employee social security numbers and pay rates.

Then she replies to Bob: "Payroll has SSNs. I'm closing it off and checking who else has grants like that."

Check Who Else Can Read Payroll

She tries Kafka's own CLI first. Asked about the payroll topic, it lists one account, and that account is blocked:

$ kafka-acls --list --topic payroll-employee
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=payroll-employee, patternType=LITERAL)`:
	(principal=User:reporting-svc, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:reporting-svc, host=*, operation=READ, permissionType=DENY)

By default, kafka-acls --list only shows grants on that exact name. --resource-pattern-type match adds the prefixed and wildcard grants, but it still doesn't say which grant wins, which team owns each account, or what the topic holds.

So Priya types / in Ask and picks ACL hygiene watch, one of the tasks built into Console. It reads every ACL binding on the cluster and comes back with four problems, all on accounts with no application owner in Console:

• legacy-etl has All on every topic and every consumer group. It can read, write, and delete anything on the cluster.
• bi-tool reads through the one-letter prefix p. That matches six of the eleven topics, payroll included.
• reporting-svc has an Allow on payroll-employee that a Deny cancels out. The Allow does nothing.
• migration-job can write to orders-v1-legacy. That topic doesn't exist.

It's a recommendation, and the card says so: "Nothing happens unless you act on it. Dismissing it is a valid answer."

Remove the Grants That Do Nothing First

Two of the four are safe to fix straight away, because neither grant does anything. Priya asks Ask to remove both, and adds: "Take me to each one and show me what to press."

Ask opens each account's page and highlights the row to delete, then the Save Changes button. Priya presses both herself. Ask then checks the brokers, and neither account has any grants left.

Ask points at each grant, Priya removes it, and Ask checks the brokers.

That leaves the two that matter. Nobody has claimed bi-tool or legacy-etl, both can still read payroll, and Priya can't tell from their names which jobs depend on them.

See Where Broad Access Meets Sensitive Data

Before she decides, Priya checks the latest report from an agent that runs on this cluster every day. It runs a custom task written for this cluster: a short prompt and seven read-only tools that inspect schemas, sample records when a topic has no schema, and list who can read each sensitive topic.

The report puts payroll-employee at the top, readable by bi-tool and legacy-etl. It also turns up aws.identity.cdc.customers.json.v1, a schemaless copy of the customer data. The agent samples records from every topic without a schema, and in this one it found names, emails, income, and credit scores. The run's trace shows each of those reads.

The sensitive data report, its trace, and the Deny that closes payroll.

Priya has three options for the two accounts:

• Cut them back now. Payroll is safe at once, but any job still using them breaks, and nobody knows which jobs those are.
• Wait for an owner. Nothing breaks, and payroll stays readable for as long as the search takes.
• Close payroll only. A Deny on the payroll- prefix blocks the sensitive topics and leaves everything else running.

She picks the third. In Kafka a Deny always beats an Allow, so she adds Deny Read on payroll- for bi-tool, and Deny All for legacy-etl, which could otherwise still write to or delete the payroll topics. If a legitimate job reads payroll through one of these accounts, it breaks now, and Priya would rather find out that way than leave social security numbers readable while she looks for an owner.

Turn a One-Time Review Into a Weekly Agent

The review she ran from Ask ran once, in the chat, and wasn't saved. Priya doesn't want the next over-broad grant to wait for the next leak.

She presses Schedule it under the review, and Ask fills in the agent wizard: the name, the task, the model, and a weekly run on Monday at 08:00 UTC. It points at the button that creates the agent's own identity, with only the permissions this task needs, and then at Create and enable. Priya presses both.

Ask fills in the wizard, Priya creates the agent, and runs it once.

She presses Run once. This time the recommendation is saved on the Output page. The two dead grants are gone from it, both accounts are flagged as unclaimed, and it says this about her Deny: "The Deny is the only thing protecting payroll, so keep it until the wildcard Allow is removed."

Bob Fixed What He Owns. Agents Checked the Rest.

Bob narrowed the part his team owns from his own terminal. What the grant reached, and what other accounts reach, took Priya, who can see the whole cluster, and two agents that keep checking it. A person made every change. Here's what each piece does:

When Priya asks Ask who owns legacy-etl and bi-tool, the answer starts with "Neither account has an owner". Console's audit log shows who added the payroll Deny, "but nothing shows who granted the original access." Until someone claims the two accounts, the weekly agent keeps flagging them. In the next use case, Priya finds their likely owners and cuts each account back to what it uses.

See It Live on Thursday

Watch us demo use cases like this live on Thursday, 19 November, at 11:00 AM ET / 8:00 AM PT: Leave the Chores to Kafka Agents.

---

Related: If Kafka Credentials Leak, Does the Attacker Get Plaintext or Ciphertext? → · How to Manage Kafka ACLs and Security at Scale → · Introducing Conduktor Agents → · Conduktor MCP + Skills for AI Agents →