Automating Kafka Audit Evidence

For most teams, Kafka compliance gets checked once a year: in the weeks before an audit.

"This week, we also have the auditor for PCI in here, in house." — Technical operations lead at a card-payment processor

The auditor was on site that week. In the weeks before, someone had to put the answers together:

• Which service accounts read the PII topics?
• Which topics have encryption on?
• Who approved that ACL change?
• When were the credentials last rotated?

None of it sits in one place: you check broker configs (or your provider's console, on managed Kafka), search the authorizer logs in Splunk or Elasticsearch, find who approved what in Slack and Git, and pull the rest from a spreadsheet.

By the time it's compiled, the evidence is already stale: it describes the cluster three weeks ago, not today.

The problem was never that compliance is hard. It's that the evidence gets rebuilt from scratch every time.

It doesn't have to work that way. The platform can produce the evidence as a byproduct of normal operations, instead of it being a project you spin up before every audit. Here's what that takes.

What the auditor is actually checking

Auditors don't want your policy document. They want proof that the control held, for the whole period, not just on the day they ask. And the bar keeps rising on how granular that proof has to be:

"Our auditors are now going to start to look at individual access controls and the fine-grained details... we can't just boil it under a service account anymore." — Software engineering director at a logistics company

Once access has to map to individual identities, a shared service account stops being an answer. "The analytics-svc account read it" doesn't tell the auditor which human used it, under which grant, or who approved it.

Same story on change management. It's not enough that a change happened, you have to show it went through the process:

"We need some kind of audit log, so that if the PCI auditor comes and asks, I can show them the operation team is doing this change." — Platform architect on a PCI-scoped Kafka project

So the questions are predictable. The work to answer them shouldn't be:

What the auditor asks	What actually answers it
Who can access sensitive data, and who approved it?	A live access map: principal, grant, approver, last access, tied to a role, not a service account
Is the data encrypted, in transit and at rest?	Encryption status across every cluster, plus key-rotation history
What changed in production, and was it approved?	A change log: who, what, when, the justification, the result
Does retention match what the policy claims?	Per-topic retention, verified continuously, with deletion logged

None of these need a three-week investigation. They need the platform to have written the answer down as it happened.

Point-in-time compliance is a fiction

The PCI-scoped team above doesn't wait for the audit. They forward their Kafka access logs continuously: who is accessing which topic, who is changing the information on which topic. The problem is that most of it is still wired together and maintained manually.

What you want instead is continuous validation: the rules that should always hold get checked every day, and a violation surfaces the moment it happens, not the next time someone checks.

A handful of checks that should never go stale:

• Every production topic has replication factor ≥ 3
• Every topic carrying PII has encryption enabled
• Every service account had an access review in the last 90 days
• Every schema uses BACKWARD or FULL compatibility, never NONE

When one breaks, you hear about it immediately: "Topic customer-events-new created with RF=1 in production. Policy requires RF≥3. Created by team-x at 14:23." Not when the auditor later asks why it had one replica during the incident.

A stale snapshot can show encryption on when a recent misconfig has turned it off. The auditor finds the mismatch, and now every other artifact you handed over is in doubt.

Evidence as a byproduct: three shifts

Getting there is three changes in how the platform behaves.

• Evidence: today it gets rebuilt for each audit; it should be emitted. Every state-changing operation writes a structured record as it happens: timestamp, authenticated principal, operation (create_topic, grant_acl), the resource, the parameters, the result. The logs are the artifacts, so when someone asks "who accessed X?" or "what changed before incident Y?", the answer is already written down. (The audit-logging post covers this part in depth.)
• Violations: detecting a bad config after it lands is the expensive path. Policy enforcement at the proxy layer rejects the non-compliant request at creation time, which turns "the audit found violations" into "violations never landed". The RF=1 topic from earlier never gets created, because the request to make it is refused.
• Answers: finding them becomes a query. "All topics containing PII, with their retention, encryption status and access controls" is one question against the audit trail, and the answer reflects the current state, not a snapshot from the last time someone compiled it.

The auditor's questions stop being investigations and become lookups.

What each framework actually asks for

Different frameworks want different evidence, but the shape is the same: a question, and an artifact the platform already produced.

Framework	What it wants you to prove	The artifact that proves it
SOC2	Access is controlled, changes are approved and logged, incidents are handled	Access matrix with approval dates, change log with results
GDPR	Lawful basis, data minimisation, deletion honoured	Data inventory of personal-data topics, retention per topic, deletion logs
HIPAA	Minimum-necessary access to PHI, encryption, audit controls	PHI access log per principal, encryption + key-rotation report

Retention is where they diverge. SOC2 doesn't mandate a fixed number, the report covers an audit window (often a year) and you keep evidence to support it. GDPR ties retention to how long you hold a lawful basis. Financial-sector rules (SEC 17a-4, MiFID II) push records into multiple years. TLDR: keep audit logs at least as long as the longest requirement touching that data, and make the log immutable so nobody can argue it was edited after the fact.

Append-only storage, or a logging backend with retention locks, is what lets you say "this is what happened" instead of "this is what our notes say happened". That distinction is the entire point of an audit log.

Are you measuring the right thing?

These three numbers tell you where compliance stands between audits:

• Audit response time. The question that used to take days of log analysis should take a query. If it still takes days, nothing was automated, you just moved the manual work around.
• Evidence freshness. Reports should reflect the current state because they're generated on demand. The instant your "evidence" is a doc someone exported weeks ago, you're back to snapshots.
• Violation detection lag. Time from a bad change to someone noticing. With enforcement at write time it's zero, the change never lands. With a periodic scan, it's however long until the next run.

Where Conduktor fits

The three shifts map onto three things we already do:

• Emission: automated audit trails for every state-changing operation, with the who/what/when/why attached.
• Enforcement: policy enforcement at the Gateway, so non-compliant requests are refused, not flagged later.
• Query: RBAC and on-demand reporting, so "who can access the PII topics, and who approved it" is one question with a current answer.

That's the difference between weeks of audit prep and running a report.

Generate the evidence as you operate, and the audit stops being a project. The auditor can be on site, and you already have the answer.

---

Related: Kafka Security Compliance → · Audit Logging in Kafka → · What We Learned from SOC2 Type 2 →