Kafka Has Rate Limits. Nobody Turns Them On.

Ron Kapoor September 15, 2026 10 min read

Rows of wireframe data packets streaming left to right on a dark teal field. A single glowing lime gate stands across their path, and the packets on either side of it are lit while the rest stay faint.

Kafka rarely goes down from load. It goes down under its own clients: a deploy that multiplies worker count by ten, each worker holding a connection, a rebalance on every restart, a consumer committing after every record. The cluster stalls for minutes. The team adds brokers. The stalls continue.

Most of the controls that would have caught this ship with Kafka. In the connection-storm incidents we've worked through this year, the broker's connection-rate limit was unset in every one, and in more than one the platform team didn't know it existed. The rest are client mistakes no broker setting reaches, because the broker never sees them.

Kafka Goes Down Because of Its Clients

Picture a tier-zero cluster: a handful of brokers, tens of thousands of partitions. A service that deploys many times a day gets a concurrency change, and its worker count goes up by an order of magnitude. Every worker opens its own connections and joins the same consumer group. On the next redeploy, every one of them leaves and rejoins.

Adding brokers doesn't help, because the part that's saturating doesn't spread. Every consumer group has one coordinator, the broker leading its __consumer_offsets partition, and all of that group's JoinGroup, SyncGroup, Heartbeat and OffsetCommit traffic goes there. With the default RangeAssignor still first in partition.assignment.strategy, every member stops and hands back all its partitions whenever any member changes, so a rolling deploy of N workers is N stop-the-world rebalances landing on one node.

The pull request that starts this changes one number in one service. It passes review. It doesn't touch a Kafka setting, and it's the whole problem.

Kafka Has Rate Limits. Turn Them On.

Kafka ships controls for all of this. They're off by default.

On the broker

Setting	Default → start with	What it does
`max.connections.creation.rate`	unlimited → 100/s	New connections per second, per broker or listener (Kafka 2.7+)
`max.connections.per.ip`	unlimited → 500	Concurrent connections from one address
`max.connections`	unlimited → 300000	Total connections per broker
`request_percentage` quota	none → 200	Share of request-handling thread time per client; 100 is one thread's worth

In the client

Setting	Default → start with	What it does
`reconnect.backoff.max.ms`	1000 → 10000	Longest pause between reconnect attempts
`retry.backoff.max.ms`	1000 → 10000	Same, for request retries
`group.instance.id`	unset → set on every consumer	Static membership: a restart doesn't trigger a rebalance
`partition.assignment.strategy`	`RangeAssignor` → `CooperativeStickyAssignor` first	A rebalance no longer stops the whole group

Treat the starting values as exactly that, and size the real numbers from your own connection counts and request rates. Kafka's quotas design covers how the broker computes the delay when a client goes over, and the options explorer lists every default.

The broker-side settings all govern load: how many connections, how many bytes, how much CPU. The broker can see all of that, so it can meter it. The client-side settings govern configuration, and they live inside each application, set by whichever team wrote it, in whichever language they used.

The Broker Can't See Client Configuration

A broker quota can't tell you that a producer sent acks=1. It accepts the write and the durability guarantee silently isn't there; min.insync.replicas only matters if the producer asked for acks=all in the first place. It can't require group.instance.id; nothing on the broker knows whether a joining consumer set one. It can't limit OffsetCommit for one group; the request quota sees a principal's total CPU share and can't tell a commit from a fetch.

These are decisions the client made before the request left the application. By the time the broker sees the request, its only choices are to serve it or not, and for settings like these it has nothing that refuses a request because of how the client was configured.

That is what a proxy is for. The broker already has the limits that protect it from load. What it lacks is a place to enforce how clients behave, once, for every application regardless of language, and to log why. A Kafka-protocol proxy sits where every request passes and reads the settings the broker can't.

Four Policies, Four Client Mistakes

Each of the four policies below catches a client doing something that looks fine from inside the application and is expensive from inside the cluster. In every case the broker accepts the request as sent, because nothing in its configuration lets it do otherwise. These are the policies in Conduktor Gateway that map onto the incident at the top of this post, and every violation from any of them lands in Gateway's audit log with the identity that sent the request, the policy it broke, and why.

A producer that skips durability

When a producer writes to Kafka, it chooses how much confirmation it wants before treating the write as done. That choice is the acks setting. With acks=all, the leader broker waits until every in-sync replica has the record before acknowledging. With acks=1, it acknowledges as soon as it has the record itself, while the copies are still in flight. Faster, and if the leader dies in that window the record is gone.

The mistake. acks=1 or acks=0 on a topic that matters, or uncompressed batches on a high-volume one.
Why the broker can't catch it. min.insync.replicas only applies when the producer asked for acks=all; a producer that asks for acks=1 gets acks=1. The broker can recompress what arrives but can't refuse an uncompressed batch.
What the policy does. Makes both settings mandatory for a given application. A producer that sends acks=1 gets a PolicyViolationException naming the problem: invalid value for 'acks': 1, valid value is one of -1.
Rollout. Scope it to one service account, a virtual cluster, or everything. Start with action: INFO, which logs every producer that would have been rejected without rejecting any, then switch to BLOCK.

apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: guard-on-produce
  scope:
    username: payments-app
spec:
  pluginClass: io.conduktor.gateway.interceptor.safeguard.ProducePolicyPlugin
  priority: 100
  config:
    acks:
      value: [-1]
      action: BLOCK
    compressions:
      value: ["ZSTD", "LZ4"]
      action: BLOCK

A consumer group that rebalances on every restart

Consumers in the same group divide a topic's partitions between them, and whenever a member joins or leaves, the group pauses and redistributes. That pause is a rebalance. By default a consumer that restarts counts as a new member, so a rolling deploy of six consumers is six rebalances, and during each one nothing in the group is reading.

Kafka's fix is static membership: a consumer that sets group.instance.id to a stable name is recognised when it comes back, and the group holds its partitions for it as long as it returns within session.timeout.ms (45 seconds by default). It's been in Kafka since 2.3, and it's off unless the application sets it.

The mistake. No group.instance.id, so every deploy is a full rebalance.
Why the broker can't catch it. A JoinGroup request either carries a group.instance.id or it doesn't, and the broker accepts both.
What the policy does. Accepts only the first kind. A JoinGroup without one is rejected.
Caveat. The JoinGroup response has no field for an error message, so the consumer sees a generic failure and the reason appears in Gateway's audit log. Run it in INFO mode first and that audit log is a list of every consumer in the fleet not using static membership, which is usually the most useful output of the whole exercise.

apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: enforce-static-group-membership
spec:
  pluginClass: io.conduktor.gateway.interceptor.safeguard.ConsumerGroupPolicyPlugin
  priority: 100
  config:
    groupInstanceId:
      value: .*
      action: BLOCK

A deploy that never stops rebalancing

Every consumer group has one broker acting as its coordinator, and every rebalance runs through it. When a deployment is crash-looping or a large group is rolled quickly, rebalances arrive faster than the coordinator can finish them, which is the pile-up in the timeline above.

The mistake. A fast or unstable deploy that rejoins the group faster than it can settle.
Why the broker can't catch it. Its request quota slows a client that uses too much CPU, but it can't tell a JoinGroup from a fetch, so it can't target rebalances.
What the policy does. LimitJoinGroupPolicyPlugin caps JoinGroup requests per minute, with a separate limit for each consumer group, so one team's crash loop can't spend another team's allowance. Size it to the group: under the classic protocol every membership change makes every member rejoin, so 50 to 100 a minute suits 10 to 20 consumers.
Caveat. It watches the classic protocol's JoinGroup. A consumer on the KIP-848 protocol sends ConsumerGroupHeartbeat instead, which it doesn't see.

A consumer that commits after every record

After a consumer processes records, it commits its position so a restart picks up where it left off. By default that happens every five seconds, about seventy commits a minute for a group of six. Some applications turn auto-commit off and commit after every record so that a crash never replays more than one, and a consumer handling a few hundred records a second is then sending a few hundred commits a second.

The mistake. enable.auto.commit=false and a commit inside the poll loop.
Why the broker can't catch it. Every commit is a write to __consumer_offsets on the coordinator broker, the same one handling the group's rebalances, and the broker has no per-group limit on them.
What the policy does. LimitCommitOffsetPolicyPlugin caps OffsetCommit requests per minute, per consumer group. It never rejects a commit; it throttles, and the wait grows with how far over the limit the client is. A slightly chatty consumer barely notices, a runaway one is slowed hard. Twenty a minute leaves a healthy small group untouched.

Together, the two limits change what the coordinator spends its minute on:

Where to Start

Turn on the broker's limits. max.connections.creation.rate and a request quota. They protect the broker from load; connection-rate limits have been there since Kafka 2.7 and request quotas far longer.
Fix the client defaults that should have been defaults. group.instance.id on every consumer, CooperativeStickyAssignor first, backoff caps well above one second.
Enforce the client settings the broker can't see, at the proxy. Start every policy in INFO, read the audit log for a week, then move to BLOCK. The teams you'd have had to convince one by one never change anything.

Adding brokers treats the symptom. Central traffic enforcement is the remedy.