How to Approach Multi-Tenancy in Kafka

Is there a way to achieve multi-tenancy in Kafka? Yes! But it may not be the way you'd immediately expect. Vanilla Kafka has no concept of a "tenant". There are no namespaces baked into the protocol, and no boundary a client can belong to.

The Apache documentation itself defines multi-tenant "user spaces" as a topic naming structure combined with standard access control. For now, multi-tenancy in Kafka is something you build yourself or adopt through additional tooling. There are three approaches:

1. Prefixes, ACLs, and quotas
2. A cluster per tenant
3. Virtualization at a proxy layer

TL;DR: If your tenants share by default (normal internal teams), stay on one shared namespace with ACLs, ownership, and quotas. If your tenants should never see each other (environments, subsidiaries, partners), give each one a virtual cluster. Only buy a separate physical cluster when there's a capacity or regulatory reason.

Multi-tenancy with prefixes, ACLs, and quotas

Everything Kafka gives you natively for multi-tenancy is in the Apache docs, and it amounts to six mechanisms:

• Hierarchical topic naming. Define "user spaces" through a naming convention like .... The prefix is the tenant.
• Prefixed ACLs (KIP-290). Grant each principal access only to resources matching its prefix, so one rule covers a whole tenant.
• CreateTopicPolicy (KIP-108). A broker-side plugin that rejects topic creation outside the naming rules, so the convention is enforced at creation and not just documented.
• Per-tenant authentication. Separate credentials per tenant, so ACLs have a principal to bind to.
• Quotas. Produce and fetch byte rates, request-time percentage caps on broker threads, and controller mutation rates (KIP-599), keyed by principal or client id, plus broker-level connection limits (per broker, per IP). Real, useful, and narrower than they sound; the limits are covered below.
• Tenant-scoped monitoring. Per-topic size and throughput metrics, rolled up by prefix. Kafka has no chargeback mechanism: the metrics are raw material, and attributing cost to a tenant is tooling you build on top.

A tenant boundary then looks like one prefixed ACL:

kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --allow-principal User:teamA \
  --producer \
  --resource-pattern-type prefixed --topic teamA.

This works at moderate scale with strong automation, and plenty of organizations run shared clusters exactly this way.

Where prefixes and ACLs fall short

Everything in that list makes Kafka behave as if namespaces exist. None of it makes them exist. The difference surfaces at four points.

• Two tenants can't both have a topic called orders. There's one global name table. Every application must know its full prefixed name, which means the convention lives in application code and configuration, duplicated per environment. The day two tenants need the same name, there's no answer. Acquisitions make this concrete: both companies have a customers topic, and topics can't be renamed, only recreated and migrated.
• Every tenant sees every name. ACLs deny access, but a metadata request still returns the full topic list. Tenant A can't read teamC.hr.salaries or mna.project-neptune.events, but it can see they exist, and topic names carry information. The model is deny-based: everything exists, and one mistyped ACL exposes it.
• Renames and convention changes are inevitable. Reorgs and acquisitions collide with the fact that a prefix is baked into stored data, consumer offsets, and every client config. What started as orders becomes bu1-prod-orders-v2, and every project carries that history forever.
• Governance is one-size-fits-all. The cluster has a single authentication surface, so every tenant adopts whatever SASL mechanism or mTLS setup the brokers speak, whether or not it matches how that team authenticates everywhere else. And beyond "can this principal read this topic," there's no per-tenant policy: no way to require compression, acks, or idempotence from one tenant's producers, validate their data quality, or encrypt their fields differently from a neighbor's. Those rules live in wikis and code review, not on the wire.

🚫 "We have quotas, so we're multi-tenant."

Quotas are the closest thing Kafka has to native isolation, so let's be precise about their coverage. Byte-rate quotas throttle a principal's produce and fetch throughput, request-percentage quotas cap its share of broker request-handler and network thread time, and controller mutation quotas cap the rate of topic creations, deletions, and partition adds. What they don't do:

• No storage quotas. There's no way to cap how much disk a tenant consumes. Retention is a per-topic setting, not a per-principal budget, so a tenant with permissive retention configs grows unbounded.
• No cluster-wide budgets. Byte-rate quotas are evaluated per broker. A 10 MB/s quota on a six-broker cluster is up to 60 MB/s in aggregate, and it shifts as partition leadership moves.
• No I/O or page-cache isolation. A consumer replaying a week of history reads cold segments from disk and evicts hot data from the page cache for every tenant on the broker. Quotas throttle the next request; the expensive one has already run.
• No caps on totals. Controller mutation quotas limit how fast a tenant creates partitions, not how many it ends up with. There's no native "this tenant gets at most 500 partitions."
• Throttling, not rejection. Over-quota clients are delayed, never refused, which is right for fairness and useless as a hard limit.

A quota decides how fast a tenant goes, within one broker. It doesn't define what a tenant is, what it sees, or how much it accumulates. Set them in every model; they don't create tenancy by themselves.

One Kafka cluster per tenant

When the shared-cluster conventions hurt enough, the standard escape is physical: give each tenant its own cluster. Isolation is now absolute, names can't collide, and nothing leaks, because nothing is shared.

The cost is that every boundary is now infrastructure. Each cluster brings its own brokers (typically three to six), its own KRaft quorum, its own monitoring, patching, and capacity planning (a common estimate is around 40 operations hours per cluster per year), and often its own license. The count grows with the org chart, not with load: new project, new environment, new acquisition, new cluster.

You've also created data silos. The moment data needs to cross a boundary, you're running MirrorMaker or a replication link, which means a second copy to keep in sync, lag to monitor, and a reconciliation problem you didn't have before. The isolation you paid for is now the thing you engineer around.

Virtual clusters: multi-tenancy at the proxy layer

The third approach adds namespaces one layer above the broker. A Kafka-protocol proxy sits between clients and the cluster, and presents each tenant a virtual cluster: a view in which only that tenant's topics exist, under whatever names the tenant uses, backed by shared physical infrastructure. Conduktor Gateway implements this as Virtual Clusters.

A vCluster is a namespace rule inside the proxy. For example, physical topics prefixed qa. appear inside the qa vCluster with the prefix stripped. A QA application connects with a service account bound to that vCluster, asks what topics exist, and gets one answer: orders. Not qa.orders (the prefix is the proxy's business, not the application's), not dev.orders (different namespace), and not anything else on the cluster. When it produces to orders, the proxy writes to the physical qa.orders.

That resolves each wall from the prefix world:

• Name collisions disappear. The dev vCluster also has a topic named orders, physically dev.orders. Same application code in both environments, no prefix variables, no per-environment config drift. An acquired company keeps its own customers topic inside its own vCluster.
• Metadata is scoped structurally. Unmapped topics aren't denied, they're absent from the response. A topic that was never prefixed or aliased into the vCluster doesn't exist in it, so the failure mode shifts from "one wrong ACL exposes a topic" to "someone deliberately mapped it in."
• Names become mappings. A rename is a config change in the proxy, not a migration.
• Governance becomes per-tenant. The same choke point enforces traffic control (require compression, acks, or idempotence on produce, and rate-limit per tenant), data quality validation with a dead-letter topic, field-level encryption and masking, and throughput metering that turns chargeback from a project into a report.

When something does need to cross a boundary, you don't tear down the wall. An alias maps one physical topic into another vCluster under a chosen name:

apiVersion: gateway/v2
kind: AliasTopic
metadata:
  name: fx-rates        # the name inside the qa vCluster
  vCluster: qa
spec:
  physicalName: fx-rates

One shared reference topic, visible in two namespaces, zero copies, and no replication pipeline.

Tenants still share the same brokers, disks, and page cache; the proxy doesn't partition or schedule any of that. It virtualizes the protocol surface: names, metadata, credentials, and the policy attached to them.

Where this runs. Virtual Clusters are a Conduktor Gateway capability, and they require Gateway to manage authentication and authorization for the tenants involved. We've written before about using them to isolate non-prod environments, which is the most common first deployment.

Which approach, when

Here's the comparison in one place:

	Prefixes + ACLs + quotas	Cluster per tenant	Virtual clusters
Name collisions	Unsolved, convention in app code	Solved by separation	Solved by translation
Metadata visibility	All names visible to all tenants	Nothing shared	Scoped per tenant
Cost per new boundary	Near zero	Brokers, quorum, ops, license	A config object
Cross-boundary data	Native (same cluster)	Replication (MirrorMaker)	An alias mapping
Enforcement	Discipline + review	Physical	Structural, at the proxy

But the more important question isn't which mechanism, it's which relationship your tenants have. There are only two:

• These tenants should never see each other. Environments (dev should never touch QA), subsidiaries with regulatory separation, acquired companies with colliding names, and external partners. Sharing between them would be a mistake. Wall them off: virtual clusters, with aliases as the rare, deliberate exception.
• These tenants share by default. Normal product teams in one company, where the search team consuming the payments team's topic is the entire point of running a streaming platform. Don't wall them. Keep one shared namespace with ACLs, explicit topic ownership, and a governed request-and-approve workflow, which is what Self-service provides on top of the same Gateway and Console.

Frequently asked questions

If you want to see Virtual Clusters against your own cluster layout, talk to us.