Conduktor works with banks, payment processors, card networks, and digital banks that run Apache Kafka. The data that moves money has moved onto Kafka, and the controls that keep it correct and access-governed are often spread across many application teams and multiple clusters.
Many reimplement their own field encryption, deduplication logic, and schema checks. When the platform team faces an audit, it has no unified visibility, because that logic and enforcement sit in the wrong place.
This paper is about an operating model, one we call the Kafka control plane, that puts that enforcement in one central place, so a platform team can prove correctness, retention, and access to an auditor.
It is written for the person who owns the Kafka estate inside a regulated fintech: the Platform or Streaming Lead. It is meant to be co-signed by the people who share the risk:
The CISO, who wants an independent enforcement layer over managed Kafka.
The Head of Data or DPO, who owns the tension between GDPR erasure and AML retention.
The CTO, who owns the build-versus-buy decision.
Scope. By transaction and payment data we mean the streams that move money and prove it moved: payments, ledger entries, settlement, reconciliation, KYC, and audit streams. Low-latency trading and market data are out of scope, because they play by different rules (deliberate duplication to win a race, disposable intermediate state because only the latest price matters).
Note that Kafka cannot be the ledger: funds reservation, overdraft handling, and balance invariants need linearizability and belong in a transactional store. The control plane owns only the delivery-layer correctness.
For a listed fintech, or a subsidiary of a US-listed parent, there is a further layer: the moment these streams feed financial reporting, the streaming platform falls inside SOX IT general controls (ITGC), namely change control over what shapes the data, logical access over who can touch it, and IT operations over retention and recovery. SOX is mostly an organizational program of controls and ownership that no tool provides; what a control plane can provide is the ITGC evidence an auditor asks for.
This data is on Kafka because of what it provides:
Outbox and change-data-capture. Get a committed state change reliably onto a topic.
Event sourcing. Make the log the source of truth that state is derived from.
Audit trails. Append-only records of what happened, who did it, and why.
Webhook ingestion. Land external transaction events on a topic before anything internal touches them.
In each of these the streaming layer is the system of record, not a buffer. Kafka has become the ledger of record for delivery, even when the authoritative balance lives elsewhere.
"Accountants don't use erasers." — an old accounting adage
Finance has always worked this way. You never alter a posted entry, you add a correcting one, so the mistake and its fix both stay on the record for an auditor to follow. An append-only log is that same discipline expressed as a data structure, which is why correcting a reconciliation break with an UPDATE in a database is a fast track to a compliance finding. Kafka hands you the append-only structure for free; the guarantee that no one can quietly rewrite it is the part you still have to enforce.
"Of course we need that, because our Kafka usage is for the payment and transaction of data." — Platform engineering lead, digital retail bank
When Kafka carries transaction and payment data, every application team ends up re-implementing that correctness work on its own. One team writes its own field-encryption wrapper, another its own dedup barrier and schema checks in consumer code, and others mask PII before logging.
The platform team handles the ACLs and the audit burden centrally, and cannot push all the controls it needs down to the application teams: that is too scattered and creates too much friction. So the work ends up half in the apps and half on the platform, inconsistent and hard to prove.
To run Kafka inside its PCI-DSS zone, Afterpay's payments platform built three controls into client code, because standard Kafka clients do none of that:
Card-number detection and obfuscation into a custom Avro serializer.
HMAC message-integrity verification against a KMS key, into custom producer and consumer libraries.
A chain of PrivateLink hops to keep cardholder data away from teams that should not see it.
It's the per-team model this paper is about: every one of those controls lives in client code that each team has to build, ship, and maintain, and the next team that touches it starts over.
A fintech often runs a mix of self-managed Apache Kafka (on premises), AWS MSK, Confluent Cloud, or another flavor, each with its own access model, audit output, and encryption practice. There is no single place to apply one policy or prove one control. Governance is partial, and full of unknown unknowns.
"And then we got derailed by all the audit stuff that we have to comply with and the findings. Right now March is all about compliance and getting our audit stuff in place for the entire team." — Platform / security manager, payments processor
Compliance should not be a one-off task; it should be continuous. Today, when compliance work arrives, it is scattered across clusters and apps, and the platform team generally loses a big chunk of its roadmap to assembling the evidence by hand.
Money software reduces to three principles, the ones the Fintech Engineering Handbook by Voytek Pituła names: no invented data, no lost data, no trust. The first two are about the data. The third, no trust, is about everyone who touches it: verify everything, and trust no source by default, not a provider, not another service, not your own operators. On Kafka, each one becomes something to enforce:
No invented data means idempotency.
No lost data means durability and erasure on an immutable log.
No trust means controlling who and what can touch the streams.
The way Kafka is governed today is generally not enough, because enforcement is either incomplete or in the wrong place:
Kafka ACLs are coarse and insufficient. Broker ACLs and authorizer logs record authorization grants and denials, not per-record consumption. Consumer-group offsets show progress, not identity-attributed reads. PCI Requirements 8 and 10 expect an audit trail that ties every access to cardholder data back to a named identity, so the question they force, who actually read this PAN (the cardholder's card number), is unanswerable at the data layer.
You need identity-bound access to answer who is authorized and who is connected. DORA imposes similar access-control and audit-trail obligations. Shared service-account credentials, which are often the norm in Kafka clients (and in tooling that reuses existing ones), break the unique-ID attribution that PCI requires: several humans and tools hide behind one principal.
This is also the classic SOX finding on a streaming estate: a service account nobody can identify still has write access to the general-ledger integration, and no one can say who granted it or who uses it. It is a segregation-of-duties gap in ITGC terms, a producer identity with an uncontrolled write path into financial reporting, and it is unanswerable at the broker, where authorization logs record grants and denials but not the human behind the principal.
No schema enforcement. Any producer with write permissions can put a malformed or unbalanced transaction record on a topic, or serialize an amount as a bare JSON number (an IEEE-754 double, so money crossing the wire that way silently loses precision). Even when you have a Schema Registry, it does not enforce anything on the wire.
Operators are Gods. Anyone with produce or admin rights can delete records, write tombstones, or change retention. An audit trail they can edit proves nothing to a regulator.
At-least-once means duplicates are guaranteed. A retry, a consumer-group rebalance, or a consumer rewinding its offset all redeliver events you have already seen. Exactly-once delivery across system boundaries (an external PSP, a webhook) is impossible, and Kafka's own exactly-once semantics cover only Kafka-to-Kafka flows, which is precisely why an application-level idempotency pattern is still needed end to end. If the dedup barrier does not sit somewhere shared, every consumer has to reinvent its own idempotency check, or risk applying a money-moving effect twice.
Encryption at rest and TLS in transit are necessary and not sufficient. PCI Requirement 3 is explicit that disk-level encryption alone does not protect stored PAN, and Requirement 4 expects strong transport encryption even on internal segments. At-rest encryption is transparent to the broker, so managed Kafka serves the data in whatever form the producer wrote it, plaintext the moment it is served: to an authenticated client, to a leaked credential, and to the provider's own break-glass operator inside their proxy. At-rest encryption protects a stolen or decommissioned disk, not the running broker, and pushing encryption client-side instead just moves the key-management burden onto every application team.
Encryption keeps the PAN in scope; only tokenization takes it out. A field-level-encrypted PAN is still a reversible representation of the PAN, so the PCI SSC treats the ciphertext as cardholder data in scope: encrypting the topics protects the data but does not remove the brokers and consumers from the CDE.
Tokenization is different. It replaces the PAN with a token that has no mathematical relationship to it, the mapping held in a segmented vault, so a topic carrying only tokens holds no cardholder data. Do it before the produce, and the log and every consumer downstream fall outside the CDE, subject to your QSA's scope determination. That shrinks a SAQ D assessment; it does not make you SAQ A eligible, since you still ingest the raw PAN at the edge. Format-preserving tokens keep schema validation and downstream joins working on a value shaped like a PAN, and what stays in scope is the ingestion edge and the vault itself.
"From my understanding, you're encrypting the data for your client using Conduktor before it hits Confluent Cloud?" — Integration architect, retail & commercial bank
That is the move: protect the data before it reaches the managed cluster, without making every app own an encryption library.
Erasure. A GDPR request asks you to erase from a system that cannot be edited, where the data is replicated, sitting in long-lived compacted topics or topics with infinite retention (especially with Kafka Streams or ksqlDB), in tiered storage, copied across clusters, and duplicated by every downstream application into its own topics and databases.
A common misconception: Kafka is not just a temporary buffer. Whether you like it or not, how it is used naturally leads to data being stored there indefinitely, whether through infinite retention (the full history stays) or compaction (the latest value per key stays), so anyone can sweep a topic and extract the current state of every entity still on it.
Meanwhile AML requires record-keeping, and for crypto-asset service providers, MiCA requires that the same personal data be kept. A per-app deletion script cannot resolve a retain-versus-delete conflict on an append-only log.
The operating model is this: move correctness, durability, and trust enforcement off every application and every cluster's native tooling, into one layer that applies to all clients and all clusters. That layer is an inline proxy every producer and consumer connects through, sitting in front of Kafka, paired with a governance console. The proxy enforces policy on the wire; the console is where that policy and the audit trail live.
Trust no client, including your own engineers, operators, and partners. Enforce at the wire level, where no one can bypass it.
Kafka can deliver the same transaction event more than once. A retry, a rebalance, or a replay sends it again, and the day one consumer mishandles it, a payment goes out twice or a balance is credited twice, the kind of error a customer and a regulator will complain about.
Often each team writes their own protection against this, but one miss or weak implementation is enough to let a duplicate through. The deduplication itself still happens in the consumer, since it needs shared state the proxy does not hold. What the proxy enforces is the one precondition every consumer depends on: a single rule, for every producer, that every transaction event must carry a stable idempotency id. The estate then deduplicates on one guaranteed key instead of each team hoping every producer set one.
The proxy's job is to guarantee this id is always present, so a redelivered event is recognizably the same event.
Ordering is the same story: Kafka preserves order only within a partition, so an account's events stay ordered only when they share a partition key. Process a debit before the credit that funds it and the customer drops into an overdraft they were never actually in; this is why Monzo treats per-account order as a correctness requirement. A producer that omits the key lets the default partitioner scatter an account's events across partitions, and the wrong-order effect follows.
The proxy can require a stable partition key on every transaction event, exactly as it requires the idempotency id, so same-key events land on one partition and Kafka holds their order. That is the precondition, not the whole job: keeping one in-flight event per key on the consumer side stays application logic, the same split as deduplication. The proxy guarantees the key is present; it does not itself reorder.
A record only counts as durable if it reaches the log in the first place, and stays readable for as long as the regulator requires.
On ingestion. The proxy enforces that records arriving from the outbox or CDC path carry the schema and the stable id the rest of the system relies on.
Over time. The proxy enforces that a record actually uses a schema and that its payload matches the schema the record declares, so every transaction event lands well-formed and self-describing, which is what event sourcing, audit trails, and AML and MiCA multi-year retrieval rely on.
With the outbox pattern, the atomicity between the ledger and the event lives inside the transactional store, out of the proxy's sight. The proxy does not reproduce that guarantee, and it cannot detect a change that was committed to the database but never published, because it only ever reasons about what passes through it. What it does guarantee is the shape of every event that does cross the delivery layer: the idempotency id is present, the partition key is present, the schema is valid. The store keeps the atomicity, the proxy keeps the delivery contract.
A transaction event also carries multiple business dates:
Value date
Booking date
Settlement date
These are separate fields. None of them is the default Kafka record timestamp the client adds when sending a record (wall-clock time). The proxy can require those fields to be present, so a producer cannot omit them or quietly substitute the record timestamp, which would confuse the rest of the ecosystem.
Erasure on an immutable log is the hard part, and it is exactly where the DPO's two regulators pull opposite ways: GDPR Article 17 says erase the subject, while AML and MiCA say retain the record. The data is append-only, replicated, sitting in compacted or infinite-retention topics, copied into tiered storage and into every downstream sink, so a delete script can neither reach all of it nor satisfy both rules if it did.
The outcome worth engineering for is narrow and specific: erase the identity, keep the transaction. Two moves get you there.
Keep PII off the log where you can. Reference each subject by an opaque internal identifier, and hold the PII in a separate, mutable store that can be redacted on request without touching the log.
Crypto-shred the rest. For the identifying fields that have to travel in the event, encrypt each subject's fields under a key unique to that subject, which is crypto-shredding. Erasure becomes deleting that one key. The ciphertext stays on disk but is unreadable everywhere it was copied, including backups and tiered storage, as long as no copy of that key survives in a cache or a KMS backup and no consumer re-persisted the decrypted value downstream.
Crypto-shredding on the log: encrypt each subject's identifying fields under a per-subject key, and erase by deleting that key. The identifying fields become permanently unreadable, including in backups and tiered storage, while the pseudonymous transaction survives for AML and MiCA retention. Every other subject is untouched under its own key.
Deleting one subject's key leaves the pseudonymous record standing: the amount, currency, dates, and opaque identifier are still readable, so AML and MiCA keep the transaction they require, while the person behind it can no longer be identified. Two regulators satisfied by one action, and every other subject's records untouched under their own keys.
The erasure has to survive a replay. Backfills are routine, you rerun history to fix a consumer bug or recompute a model. A replay reads the same log the live traffic reads, so once a shredded subject's key is gone, the identifying fields come back as ciphertext to the replay consumer exactly as they do to a live one, and stay unreadable. Because the key is destroyed at the source, not filtered downstream, no replay path reconstitutes an erased identity.
That is the business outcome. "We cannot delete from Kafka" is precisely the storage-limitation gap an auditor writes up, and over-retention is a finding in its own right. Crypto-shredding converts that gap into a control you can demonstrate, and brings erasure-request turnaround inside the one-month GDPR window instead of an open-ended manual sweep across clusters and sinks.
The honest parts stay honest. One key per subject means millions of keys with their own custody, availability, and recovery story, plus a safeguard against shredding a record still under retention or legal hold. The master key sits in a customer-controlled KMS, so the erasure decision stays with you rather than a vendor. And ciphertext can still count as personal data while the key exists, so validate the approach with your DPO and supervisory authority.
"That is also something we are looking at, as we need to segregate PII data versus non-PII data." — Platform architect, digital-first bank / lending fintech
Kafka's native trust surface is too weak for transaction and payment data, so the proxy adds what's required:
On the way in. The proxy asserts the shape money must take, rather than only rejecting known-bad values. An amount must be a string or an integer in its smallest unit, never a bare JSON number that is silently an IEEE-754 double, and a currency code must come from a controlled set, so a typo'd or attacker-supplied code is rejected at the wire before it enters the log. External, webhook, and CDC data is held to the same rule and rejected when it does not conform. This is also the point to tokenize a PAN: replace it at the wire, before the produce, with a token that has no mathematical relationship to the card number, so the log itself never carries cardholder data, instead of merely encrypting a value that stays in PCI scope.
On the log itself. Whatever the raw Kafka ACLs allow, the proxy denies deletes, tombstone writes, and destructive admin on ledger and audit topics, so append-only is enforced rather than assumed.
On access. Replace flat native ACLs with audited RBAC and least-privilege roles over topics and consumer groups. Add the controls regulators expect: scheduled access recertification, and four-eyes on sensitive admin such as topic deletion, permission changes, replay, and retention changes. These are the same controls SOX ITGC names under logical access and change control when the topics feed financial reporting: least-privilege on who can write, recurring access reviews, and segregation of duties so the person who ships a producer does not also hold an uncontrolled write path to ledger topics. A replay is itself an identity-attributed action in that trail, so you can show an auditor who reprocessed which topic and when.
On the way out. Mask and decrypt per reader at the wire, so a downstream service receives masked PII while an authorized service sees cleartext, with no client code change on either side. An ops or support engineer browsing a topic in a console gets the same protection through console-level masking.
"That's where we're putting in Conduktor, to confirm who's coming in, and control and ensure that they have direct access only to their data. So in case an audit does come, we can guarantee that we are showing that this is what it is." — Platform / security manager, payments processor
Your operating model becomes:
One plane across vendors. The same model applies across Confluent, MSK, Aiven, Redpanda, and self-managed Kafka from one plane, so there is one place to prove compliance instead of one silo per vendor.
Governed self-service. Application teams request topics or connectors and own their data within platform-defined guardrails, which lets the platform team delegate permissions and governance to the teams without losing the control that answers an audit.
Let's see the flow: take a card top-up, where money is coming in. Duplicates still have to be caught, since the provider can redeliver the same "captured", but the distinctive hard part here is trust: you cannot take the outside world's word for what happened.
A card top-up split across two lanes: the webhook ingester verifies the signal and queries the provider for truth, the proxy guarantees every event that lands is keyed, schema-valid, and append-only, and the ledger does the posting arithmetic neither of them touches.
The deposit process:
The user submits the deposit, and the request carries an idempotency key (because the submit can be retried)
The payment provider authorizes the card and places a hold (a reservation). The money is not yours yet, so nothing is credited.
The provider's webhook calls to say the payment was captured. The ingester verifies the signature of the payload and queries the provider's API to validate (the webhook only triggers the signal, it's not a source of truth). When the ingester produces the events, the proxy is what guarantees there is a stable idempotency id (so a re-delivered "captured" deduplicates to a single credit downstream) and that it matches its declared schema, before any downstream app sees it.
Once the capture is confirmed, the money is in flight, captured by the provider but not yet settled to your bank, so it posts through a clearing account for the time being.
Proxy vs ledger. The proxy never does the accounting itself; that happens in the ledger. It does require the provider's transaction id on each event, so the later reconciliation is a key-join rather than a guess from amount and time.
What happens next plays out over days and weeks.
Settlement. Days later the provider settles one transfer covering many deposits, and reconciliation matches that batch against the clearing account, one settlement to many transactions.
Chargeback. If the cardholder disputes the payment weeks later, the chargeback is a new compensating event that references the original posting, not an edit to it.
Access. Throughout, the cardholder's data is encrypted on produce and masked per reader, so an ops engineer reconciling the batch never sees a raw card number.
This flow is about ledger logic and protecting the system: a duplicate delivery, or an operator's eyes, could invent, lose, or leak that data, which is why a proxy is required to enforce the necessary controls: idempotency id, valid schema, append-only, per-reader masking, and so on.
The card flow ends at reconciliation, the moment the processor's report, the bank's settlement file, and your own ledger have to agree.
The breaks are usually discrepancies across systems:
Timestamps that disagree (wall-clock vs value date)
An amount that lost a cent to a float
A description truncated differently on each side
The same event counted twice (missing idempotency id)
Each one surfaces as an exception a team has to clear by hand.
Most of those issues are data problems. You can only enforce data on the side you control, the events on your Kafka log. This is why enforcing data at the wire keeps your side of the reconciliation from ever being the cause of a break:
A required, well-formed provider transaction id and idempotency id on every event, so your log always carries the key the reconciliation joins on, instead of falling back to matching on other fields like amount and time.
Value, booking, and settlement dates as distinct, required fields, so the log never collapses three dates into one ambiguous timestamp. It is still on the producer to put the right value in each.
Amount as a minor-unit integer, currency from a controlled set, schema-validated types, so no float sneaks into an amount field and no mis-typed or missing field reaches the log.
Append-only audit on the Kafka side, the who, what, and when to explain the origin of a discrepancy.
A required, stable deduplication key on every event, so your consumers have one key to deduplicate on. The proxy enforces that the key is present, the consumer does the actual dedup.
This disciplines the Kafka side, and it makes investigations simpler: if the numbers disagree, you know the break came from an external source, not from your own log.
Every producer and consumer of that data passes through a Kafka proxy (data plane), and a Console (control plane) to govern every cluster behind it. The per-app encryption and per-cluster controls are enforced, so the same policy applies across every provider without any client code change.
Producers. Applications, webhooks, CDC connectors reading a database write-ahead log, and external API calls. All of them carry transactional data or PII.
Consumers. Downstream services and humans (ops or support). An authorized service receives cleartext; an ops user or a generic downstream service receives masked PII.
The proxy. Every byte passes through it, to enforce the principles above:
Validates and rejects on schema, checks message integrity, enforces the idempotency id, and rate-limits per producer so a runaway job cannot starve a payment topic's throughput.
Marks ledger and audit topics append-only and denies destructive operations, then applies field-level encryption keyed per subject, feeding a crypto-shredding keystore. A single master key sits in a customer-controlled external KMS off to the side.
Masks and decrypts per reader.
A unified governance model spanning every cluster. Visible in Console, it holds cross-provider RBAC and ownership, the self-service request flow with guardrails, the access-grant audit trail, and consumer-lag and delivery observability. Config-as-code feeds both the console policies and the proxy enforcements. The audit stream is exported to the SIEM you're using.
"I've been chasing audit logs, trying and trying to get them into Splunk for all of our products." — Platform engineer, payments processor
This user had a complex ecosystem with multiple Kafka clusters and vendors, and no proxy in front of it.
Architects may be skeptical at first (isn't a Kafka proxy a single point of failure?) until they remember they have dealt with proxies forever in the HTTP world: API Gateways, reverse proxies, nginx, traefik. A proxy is still a real decision, because every byte passes through it, which makes it an availability and latency question.
A Kafka proxy is not a dumb load balancer. Kafka embeds broker addresses inside its own metadata responses: a client bootstraps, gets the full broker list, then connects directly to each partition leader. A Kafka proxy rewrites those advertised addresses to ensure clients always go through it.
As with an HTTP reverse proxy, clients must talk only through it for the controls to apply; a Kafka proxy is the same, so you expose one address and hide the many brokers behind it. Reaching a managed cluster across a network otherwise means a load balancer and a firewall opening for every broker, redone each time you add one; behind the proxy, a new team points at a single endpoint and the cluster's own network is never touched.
As with any proxy, a few questions need to be settled first:
What happens when a proxy fails. You always run multiple proxies for resiliency, so even if one fails, or you roll-upgrade them, traffic keeps flowing. It's like Kafka itself: you can restart a broker and the system auto-adapts. Kafka proxies follow the same pattern: because they rely on Kafka to hold their state, Kafka stays your central point of failure, not the stateless proxies in front of it.
A Kafka proxy also simplifies failover. Because clients only ever know the proxy's endpoint, failing a cluster over to its secondary is a single API call that reroutes all traffic with no client change or redeploy, which makes the failover reproducible and testable, useful for regulations like DORA that require continuity testing. It also removes the cross-team coordination that is usually one of the most painful parts of a DR or failover.
How it scales. Proxies are stateless: instances run side by side in front of the Kafka clusters, sized to the throughput going through them and the use cases they enable (routing, verifying, altering payloads, and so on).
How it protects the payment path from a noisy neighbor. On a cluster shared across teams, a runaway producer, typically a batch or analytics job, can saturate broker network, CPU, or IO and push the authorization path past its latency budget. Kafka's native client and broker quotas are the first line here: byte-rate and request-rate ceilings set per (user, client-id). But they are static, set by hand and revisited rarely, and they know nothing about which topic is on the critical path. The proxy adds a second, finer control: per-producer and per-consumer rate limits enforced on the wire and scoped to a tenant or a virtual cluster, so the batch job's throughput is capped before it starves the payment topic's SLA.
As with a reverse proxy, you start small in a simple passthrough mode, then add controls:
Start in Kafka passthrough: it is just routing, changing the bootstrap servers the apps point at.
Introduce policies to observe and understand your Kafka traffic: see how apps talk to Kafka, and spot bad practices or even application bugs (over-committing, over-using transactions, and so on).
Turn enforcement on: now that you understand the traffic, talk to the teams doing odd things with Kafka (they may not even be aware), then start preventing those bad practices (the proxy will deny some behaviors from now on).
Proxy access has to be tightly secured. A proxy terminates authentication, holds the masking and decryption logic, and reaches your KMS, which makes it a high-value target. It has to be hardened, isolated, and monitored like the brokers it fronts.
Gather your security and compliance teams and work through the questions below, against your estate as it stands today. Each is something you should be able to demonstrate to an auditor, and each maps to one of the three principles, and to a regulation where one applies. Nothing here assumes you have already adopted the control plane; the point is to find the gaps.
Score each as enforced at a single control point, enforced in some apps or by convention, or not enforced. Note that "in some apps" is a gap: a control you cannot prove in one place is something an auditor will not accept.
#
Can you prove it?
Principle / regulation
1
A redelivered or replayed transaction event produces a single effect (shown with a replay test), because every event carries a stable idempotency id that consumers deduplicate on.
No invented data / idempotency
2
A change committed to your source database reaches Kafka with no silent loss, reconciled database-to-topic with alerting on any gap, and the outbox or CDC path carries the schema and stable id the rest of the system relies on.
No lost data / outbox, CDC
3
Today's code can read a transaction event written years ago, because backward and forward schema compatibility is enforced before a breaking schema is registered.
No lost data / AMLD 5y retention (MiCA scope: crypto-asset services)
4*
A GDPR Article 17 erasure request can be honored on an immutable topic, reaching backups and tiered storage, while only the AML-mandated field is retained and the rest is erased or anonymized.
Retention vs erasure / GDPR Art.17(3)(b) vs AMLD (MiCA scope: crypto-assets)
5
An amount arrives as a string or a smallest-unit integer (never a bare JSON number) and a currency code comes from a controlled set, even from a producer you don't own.
No trust / precision, schema
6
The integrity and source of an incoming webhook or external event is verified before it enters the log.
No trust / webhooks, APIs
7
No one with produce or admin rights can delete, tombstone, or shorten retention on a ledger or audit topic.
You can produce a who-did-what trail of every grant and revoke on in-scope topics, and run scheduled access recertification and four-eyes on sensitive admin.
An ops or support user can browse an in-scope topic without seeing raw PAN or IBAN, and a downstream service gets masked PII while an authorized service sees cleartext.
No trust / PCI Req 3, per-reader masking
11
The same policy and access model are provably enforced across every provider in your estate (Confluent, MSK, Aiven, self-managed), not redefined per silo.
Estate-wide / DORA third-party oversight
12
Application teams can request in-scope topics only within platform guardrails, and a request that falls outside policy is refused.
Operating model / self-service
13
Raw CDC output is schema-validated and its PII fields are masked per reader, so a downstream consumer receives masked values for the fields it is not meant to read.
No trust / outbox, CDC
14
A transaction event carries value, booking, and settlement dates as distinct fields, and none of them is silently the Kafka record timestamp (by default the producer's wall-clock at send time).
No lost data / value vs booking vs settlement date
15
Sensitive fields are encrypted before they reach the broker, so the managed cluster, a leaked credential, and a break-glass operator never see cleartext, and the keys live in a KMS you control, so you can show who can decrypt a field and revoke that access by destroying the key.
Every Kafka connection is encrypted in transit (TLS), client-to-broker and broker-to-broker, including traffic that stays inside your own network, not only the connections coming from outside.
No trust / PCI Req 4
17
Every event for one account carries a stable partition key, so its events stay on one partition and are applied in order, and a debit never lands before the credit that funds it and puts the customer into a false overdraft. The proxy enforces the key is present; per-key in-flight serialization stays in the consumer.
No invented data / ordering, partition key
18
A runaway producer, such as a batch or analytics job, cannot degrade an in-scope payment topic's latency SLA, because per-producer rate limits are enforced at a single control point and scoped to the tenant, on top of the native broker quotas.
Availability / SLA, DORA operational resilience
19
A replay or backfill of historical topics does not re-expose PII that has since been erased or would be masked for the reader running it: a shredded subject's fields come back as ciphertext, and per-reader masking applies to the reprocessing consumer exactly as to a live one.
No trust / GDPR Art.17, per-reader masking
20*
A PAN is tokenized at the wire before the produce, with a token that has no mathematical relationship to the card number, so the topics and consumers behind the proxy carry no cardholder data and fall outside the PCI CDE (the ingestion edge and the vault stay in scope).
No trust / PCI scope reduction, smaller SAQ D
21
No orphaned service account holds write access to a ledger or audit topic: every producer identity with a write path into financial-reporting streams is owned, inventoried, and traceable to who granted it and who uses it.
No trust / SOX ITGC (segregation of duties)
On item 4, the control plane executes the retain-versus-erase decision, it does not make it: whether a field is covered by the AML retention exemption (GDPR Article 17(3)(b)), and for how long, is a determination for your DPO and, where needed, your supervisory authority.
On item 20, whether tokenized topics fall outside the CDE, and how much of your estate that removes from a SAQ D assessment, is a scope determination for your QSA: the control plane enforces the tokenization at the wire, the QSA validates the resulting scope.
The cost of doing nothing is not only poor audit reviews; more importantly, it leaves you exposed to data breaches, loss of customer trust, and fines.
Over-retention. In January 2026, France's data regulator fined Free Mobile and Free a combined 42 million euros, citing storage limitation for keeping subscriber data, including IBANs, with no purge process, alongside the breach that exposed it.
Cardholder data. The UK ICO fined British Airways 20 million pounds after card numbers, expiry, and CVV were skimmed for hundreds of thousands of customers.
Misconfiguration at scale. Capital One's misconfiguration reached about 106 million people and drew an 80 million dollar regulator penalty plus a 190 million dollar settlement.
IBM's Cost of a Data Breach put the average financial-sector breach at 5.56 million dollars in 2025, second only to healthcare, while the US average across all sectors reached an all-time high of 10.22 million dollars.
DORA, in force since January 2025, adds remediation orders and supervisory penalties for financial entities. For the critical ICT third-party providers they depend on, it adds periodic penalties of up to 1 percent of average daily worldwide turnover, charged for each day of non-compliance, for up to six months.
For a listed fintech, or a subsidiary of a US-listed parent, SOX adds a different kind of exposure. A material weakness in the IT general controls over the systems that feed financial reporting is disclosed to the market, and management and the audit committee answer for it personally. An ungoverned write path into ledger topics, or an access trail you cannot produce on demand, is exactly the ITGC deficiency that turns into that disclosure.
Encryption and identity and access management (IAM) are among the largest cost-reducing factors in the report, and they are the two controls this model puts at the proxy.
The table below helps you understand your baseline, what you can/cannot measure, your gaps, and your target.
Metric
What it counts
Baseline (today)
Target
Scope
In-scope topics
Topics carrying CHD/PAN, PII, ledger, audit, KYC, or trade data; defines your PCI audit scope
# today
no target; this is your scope
PAN-carrying topics not tokenized
In-scope topics holding a reversible PAN, in clear or encrypted, that could be tokenized to fall outside the CDE and shrink the SAQ; encryption alone does not remove them (#20)
# today
0
Estate under one governance model
Clusters and providers governed by one RBAC model rather than siloed tooling (#11)
# of total
all
Coverage: protect the data
Clear-text sensitive topics
In-scope topics where PAN or PII fields are unencrypted (PCI Req 3–4; #15)
# today
0
Field-level encryption coverage
% of in-scope topics with sensitive fields encrypted before the broker (#15)
% today
100%
PII reaching unentitled readers
% of in-scope topics where a reader can get PII in clear that its role should not see (#10, #13)
% today
0
Replays re-exposing erased or masked PII
Backfills or replays that re-emitted PII already erased, or served it in clear to a reader that should have been masked, verified with a replay test against a shredded subject (#19)
# per reporting period
0
In-transit encryption coverage
% of connections on TLS, including broker-to-broker inside your network (PCI Req 4; #16)
% today
100%
Schema-validation coverage
% of in-scope topics that reject off-contract records (#5)
% today
100%
Correctness: no invented or lost data
Duplicate-effect incidents
Money-moving operations applied twice from a redelivered or replayed event (#1)
# per reporting period
0
Out-of-order effect incidents
Money-moving operations applied in the wrong per-account order, such as a debit before the credit that funds it, from events scattered across partitions by a missing key (#17)
# per reporting period
0
Lost writes (DB to topic)
Committed database changes that never reached Kafka, caught by database-to-topic reconciliation (#2)
# per reporting period
0
Schema-incompatibility incidents
Consumers breaking on an incompatible contract change (#3)
# per reporting period
0
Immutability incidents
Deletes, tombstones, or retention cuts on a ledger or audit topic (#7)
# per reporting period
0
Topics with no purge policy
In-scope topics kept indefinitely with no scoped deletion; the over-retention an auditor fines (#4)
# today
0
Evidence and identity: prove access
Non-unique identities on in-scope streams
Shared service accounts and any principal that hides several humans or jobs, so a PAN or PII read cannot be tied to one person (PCI Req 8, 10; #9)
# today
0
Access-recertification gaps
Accounts on in-scope topics overdue for recertification (#8)
# today
0
Erasure-request turnaround
Time to honor a GDPR Art.17 request across log, backups, tiered storage, and sinks (#4)
days today
within one month (Art.12(3))
DR recoverability
Days since the last successful restore test, and the gap to your RTO/RPO (DORA Art.11–12)
today
within your tested target
Audit-prep effort
Person-days, and cost at your loaded rate, to assemble streaming-layer evidence per reporting period
days / $ today
lower each period
Operational burden: the operating model
Apps carrying their own encryption, idempotency, or validation
Services with bespoke security or contract logic to consolidate
# today
lower
ACL and governance ticket volume
Access-management tickets the central team handles per reporting period (#12)
# per reporting period
lower via guarded self-service
Time to provision a governed topic
Request-to-ready with guardrails enforced (#12)
days today
lower
Payment-path SLA breaches from contention
Times an in-scope payment topic missed its latency budget because another tenant or a batch job saturated shared broker capacity (#18)
# per reporting period
0
Exposure (in $)
Estimated exposure
The IBM per-record breach cost ($160 above) applied to the PII records a single realistic incident would expose, not the entire in-scope base
$ per credible incident
falls as the coverage and correctness rows reach target
What we see in the field is often due to a lack of understanding or wrong assumptions:
Infinite retention. Kafka is not just a temporary buffer but a permanent store of regulated data (compacted topics, stream-processing state, and so on) that no one scoped for deletion.
Answering 'who owns what' takes time. Teams discover that working out who owns which topic is a painful process, and with thousands of topics it can take months.
False comfort from scanners. Schema-based PII scanners give false confidence, because producers put card numbers and emails into free-text fields the scanner never flags.
IBM's 2025 report also highlights that malicious insider attacks topped the cost table for the second year running at 4.92 million dollars. Breaches spanning multiple environments cost 5.05 million dollars against 4.01 million on premises. And customer PII was both the most-stolen record type and among the priciest at 160 dollars each.
Those are the over-privileged-access, multi-estate, and clear-text-PII gaps. Each of them can lead to an incident.
To reduce exposure and introduce the necessary controls, Conduktor provides a governance plane and a Kafka proxy for the data plane.
Every application and user reaches Kafka through Conduktor, so one control plane fronts any provider on any cloud, with no change to the clients or the clusters.
It is a Kafka-protocol proxy that applies policy on the wire with no client code change and no Kafka config change. That property is the reason the model is enforceable for producers and consumers you do not own.
What it unlocks:
Encryption. Field-level and full-payload encryption with KMS-backed envelope encryption (DEK, KEK, EDEK), so PII and PAN are protected before data reaches the broker without each app owning encryption code.
Tokenization. Vault-backed, format-preserving tokenization through HashiCorp Vault's Transform engine, so a PAN is replaced at the wire with a token that has no mathematical relationship to the card number. Unlike encryption, which keeps the value reversible and in PCI scope, tokenization is what takes the topics and consumers behind the proxy out of the cardholder-data environment, the lever that shrinks a SAQ D assessment. The ingestion edge and the vault stay in scope, so it does not on its own make you SAQ A eligible, and the final scope determination is your QSA's call, so validate it with them.
Crypto-shredding. Deleting a per-subject key makes that subject's records permanently undecryptable through Gateway, reaching backups and tiered storage where tombstones cannot. Because the field is unreadable at the wire and not merely filtered for live traffic, a replay or backfill that rereads the log sees the same ciphertext, so reprocessing history cannot resurrect an erased identity.
Per-reader masking on the wire. Scopeable by service account, group, or virtual cluster, so different readers receive differently-masked bytes. The rule keys on the reader, not on whether the read is live or a replay, so a backfill consumer is masked identically to a live one and reprocessing cannot leak PII a live read would have hidden.
Data Quality. Using CEL, JSON Schema, or EnforceAvro rules, the proxy rejects malformed or wrong-schema records, and enforces contract rules on every transaction event, such as a required idempotency id and a stable partition key (the no-invented-data rules) or a currency code from a controlled set. The deduplication and the per-key ordering stay the consumer's job; the proxy guarantees the id is there to deduplicate on and the key is there so an account's events stay on one partition and are applied in order.
Message-integrity signing and verification. On records as they pass.
Producer guardrails. Enforce the safe client configuration no matter how a team set it: acks=all with min.insync.replicas ≥ 2, so a write is acknowledged by a quorum of in-sync replicas and survives a broker loss (acks=all on its own can still ack a single copy once the ISR shrinks), and enable.idempotence, so a producer's own retries do not turn one write into two within a session. That guarantee is per-partition and per-session, which is why the durable dedup key is still the business idempotency id above, not this flag.
Throughput guardrails. Producer and consumer rate limiting enforced at the proxy and scoped to a service account, group, or virtual cluster, finer than the broker's static, topic-blind quotas, so a runaway batch or analytics job cannot starve a payment topic's latency. It complements the native broker quotas rather than replacing them.
That is how you build confidence that the data you depend on stays correct and under control.
Then you need one cross-provider plane. Console governs Confluent, MSK, Aiven, Redpanda, and self-managed Kafka from a single pane, with global cross-cluster RBAC, data policies, and GitOps, while Gateway fronts the physical clusters underneath.
On top of that sit ownership and self-service with platform guardrails, the access grant-and-revoke trail, consumer-lag and delivery observability, and audit-log export in a vendor-neutral format to an external SIEM.
For a SOX audit, this is the ITGC evidence layer for the streaming platform: an inventory of who and what can write to ledger topics, service accounts that are owned and attributable rather than orphaned, and a grant-and-revoke trail that answers the logical-access and segregation-of-duties questions on demand. Console produces the evidence; the SOX control program, the ownership, and the sign-off stay yours.
Schema Registry Proxy adds authentication, authorization, and observability on top of Confluent Schema Registry, wired into Console self-service, which supports the schema-evolution and schema-access angles of the model.
Virtual Clusters give strong tenant and partner namespace isolation, with their own authn/authz, topics, and policy scope, on a single physical cluster. Use them for subsidiary, line-of-business, and partner-zone isolation. That boundary is also a scope for throughput limits: a rate limit set per virtual cluster keeps one tenant's spike from eating another's share of the shared brokers, so the isolation covers performance and not only access. For per-internal-team multi-tenancy, Console self-service is the right primitive, not Virtual Clusters.
Automation. The CLI and config-as-code put topic, schema, and permission changes for in-scope streams through the same reviewed, version-controlled change trail as application code, so governance runs as pipelines rather than tickets, and the audit evidence assembles itself instead of costing person-days each cycle.
Insights. Consumer-lag and delivery observability, data-quality reporting, and usage visibility across the whole estate surface the metrics the self-assessment and risk matrix track, so a control gap shows up as a dashboard line before it shows up as an audit finding.
Governed AI access.An MCP server and agent skills let AI agents observe and operate the estate through the same governed path as any other client: an agent reads through the proxy, so it sees masked PII rather than raw PAN, and every action it takes is a least-privilege, audited grant. That is what makes an agent on money data safe: the control plane holds the context, so the agent is useful, and holds the permissions, so it stays in bounds.
A CTO will reasonably ask: why not build this ourselves? In isolation it can look simple: cover a few apps and producers, or set a convention you ask every team to honor, which enforces nothing. To hold a rule for producers and consumers you do not own, it has to run on the wire, in front of every client. That means a proxy.
Forking an open-source proxy is hard. The knowledge it demands is real: the protocol's constraints and behavior, the compatibility rules. The Kafka protocol is not HTTP; it is stateful and more complex, and it often takes years to get comfortable with. Kafka is a distributed system, after all. The moment you mask a field or encrypt a payload, you are decoding and recompressing record batches, and you have to keep idempotence and transactions intact through the rewrite. Then you need a serious test harness. We've published ours, have a look and see what it takes.
Compatibility is a killer. A team bumps its client library, your proxy downgrades the protocol because it does not support the new version, and the consumer group slides into a rebalance loop: no error, just lag climbing on a payments topic. You cannot reproduce it, because it depends on that client version, that fetch size, that session state, that rebalance timing. Every Kafka release and every client library is a new version of the compatibility surface you have to stay correct across.
Then you have to manage a heterogeneous estate:
One access model and one audit story across Confluent, MSK, Aiven, and self-managed Kafka, each with its own auth.
Per-reader masking and field encryption fast enough to stay inside a payment latency budget, even when your KMS hiccups.
A key lifecycle for the millions of per-subject keys crypto-shredding needs.
Four-eyes and recertification on sensitive admin.
Audit evidence you can produce on demand.
Keeping all of it correct, forever, is not a project that ends. It takes your rarest engineers, the ones comfortable with byte-level protocol, distributed-systems timing, and crypto on the hot path.
The question is not whether your team can build it. Your good engineers can. It is whether you want to:
put your best engineers on infrastructure that vendors already master and sell for a fraction of the headcount it would cost you to build and maintain;
answer to the regulator for an uncertified component sitting in your data path;
keep it alive and evolving after the engineers who built it have moved on.
Build-versus-buy here was never about the first version. Your edge was never rewriting the Kafka wire protocol.
Run back through the self-assessment above. What matters is how much of it you can demonstrate enforced at a single control point, rather than scattered across apps or missing entirely.
Where you stand
What it looks like
First move
Per-app and per-cluster
Correctness, retention, and access live in application code and native ACLs, and most of it cannot be shown to an auditor on demand.
Put a single control point in front of one in-scope topic and prove one rule end to end.
Partial
Some controls are centralized, others are still per-app, and governance reaches some clusters but not the whole estate.
Extend coverage to the clusters and providers still ungoverned.
Boundary-enforced
Most controls sit at a single control point across the estate, and the gaps that remain are known and named.
Close the named gaps and make each one provable to an auditor.
One proxy, one plane
Every principle is enforced once, and provable across providers.
Hold the line: recertify access and keep the audit evidence current.
How to produce a gap report:
Map your in-scope topics against the self-assessment.
Mark which principles are enforced at a single control point today, in some apps or by convention, or nowhere.
Turn that into a gap report: the qualitative answers, quantified by the Baseline and Target metrics above.
The Fintech Engineering Handbook by Voytek Pituła is recommended further reading: it catalogs the money-handling patterns in full, while this paper covered the operating model for enforcing the ones that land on Kafka.
Talk to us and we will help you build the gap report against your Kafka estate.
