# Kafka Has Rate Limits. Nobody Turns Them On.

Kafka rarely goes down from load. It goes down under its own clients: a deploy that multiplies worker count by ten, each worker holding a connection, a rebalance on every restart, a consumer committing after every record. The cluster stalls for minutes. The team adds brokers. The stalls continue.

Most of the controls that would have caught this ship with Kafka. In the connection-storm incidents we've worked through this year, the broker's connection-rate limit was unset in every one, and in more than one the platform team didn't know it existed. The rest are client mistakes no broker setting reaches, because the broker never sees them.

## Kafka Goes Down Because of Its Clients

Picture a tier-zero cluster: a handful of brokers, tens of thousands of partitions. A service that deploys many times a day gets a concurrency change, and its worker count goes up by an order of magnitude. Every worker opens its own connections and joins the same consumer group. On the next redeploy, every one of them leaves and rejoins.

Adding brokers doesn't help, because the part that's saturating doesn't spread. Every consumer group has one coordinator, the broker leading its `__consumer_offsets` partition, and all of that group's `JoinGroup`, `SyncGroup`, `Heartbeat` and `OffsetCommit` traffic goes there. With the default `RangeAssignor` still first in `partition.assignment.strategy`, every member stops and hands back all its partitions whenever any member changes, so a rolling deploy of N workers is N stop-the-world rebalances landing on one node.

The pull request that starts this changes one number in one service. It passes review. It doesn't touch a Kafka setting, and it's the whole problem.

## Kafka Has Rate Limits. Turn Them On.

Kafka ships controls for all of this. They're off by default.

### On the broker

| Setting | Default → start with | What it does |
|---|---|---|
| [`max.connections.creation.rate`](https://cwiki.apache.org/confluence/display/KAFKA/KIP-612%3A+Ability+to+Limit+Connection+Creation+Rate+on+Brokers) | unlimited → 100/s | New connections per second, per broker or listener (Kafka 2.7+) |
| `max.connections.per.ip` | unlimited → 500 | Concurrent connections from one address |
| `max.connections` | unlimited → 300000 | Total connections per broker |
| `request_percentage` quota | none → 200 | Share of request-handling thread time per client; 100 is one thread's worth |

### In the client

| Setting | Default → start with | What it does |
|---|---|---|
| `reconnect.backoff.max.ms` | 1000 → 10000 | Longest pause between reconnect attempts |
| `retry.backoff.max.ms` | 1000 → 10000 | Same, for request retries |
| [`group.instance.id`](https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances) | unset → set on every consumer | Static membership: a restart doesn't trigger a rebalance |
| `partition.assignment.strategy` | `RangeAssignor` → `CooperativeStickyAssignor` first | A rebalance no longer stops the whole group |

Treat the starting values as exactly that, and size the real numbers from your own connection counts and request rates. Kafka's [quotas design](https://kafka.apache.org/documentation/#design_quotas) covers how the broker computes the delay when a client goes over, and the [options explorer](https://kafka-options-explorer.conduktor.io/) lists every default.

The broker-side settings all govern **load**: how many connections, how many bytes, how much CPU. The broker can see all of that, so it can meter it. The client-side settings govern **configuration**, and they live inside each application, set by whichever team wrote it, in whichever language they used.

## The Broker Can't See Client Configuration

A broker quota can't tell you that a producer sent `acks=1`. It accepts the write and the durability guarantee silently isn't there; `min.insync.replicas` only matters if the producer asked for `acks=all` in the first place. It can't require `group.instance.id`; nothing on the broker knows whether a joining consumer set one. It can't limit `OffsetCommit` for one group; the request quota sees a principal's total CPU share and can't tell a commit from a fetch.

These are decisions the client made before the request left the application. By the time the broker sees the request, its only choices are to serve it or not, and for settings like these it has nothing that refuses a request because of how the client was configured.

That is what a proxy is for. The broker already has the limits that protect it from load. What it lacks is a place to enforce how clients behave, once, for every application regardless of language, and to log why. A Kafka-protocol proxy sits where every request passes and reads the settings the broker can't.

## Four Policies, Four Client Mistakes

Each of the four policies below catches a client doing something that looks fine from inside the application and is expensive from inside the cluster. In every case the broker accepts the request as sent, because nothing in its configuration lets it do otherwise. These are the policies in [Conduktor Gateway](https://www.conduktor.io/gateway) that map onto the incident at the top of this post, and every violation from any of them lands in Gateway's audit log with the identity that sent the request, the policy it broke, and why.

### A producer that skips durability

When a producer writes to Kafka, it chooses how much confirmation it wants before treating the write as done. That choice is the `acks` setting. With `acks=all`, the leader broker waits until every in-sync replica has the record before acknowledging. With `acks=1`, it acknowledges as soon as it has the record itself, while the copies are still in flight. Faster, and if the leader dies in that window the record is gone.

- **The mistake.** `acks=1` or `acks=0` on a topic that matters, or uncompressed batches on a high-volume one.
- **Why the broker can't catch it.** `min.insync.replicas` only applies when the producer asked for `acks=all`; a producer that asks for `acks=1` gets `acks=1`. The broker can recompress what arrives but can't refuse an uncompressed batch.
- **What the policy does.** Makes both settings mandatory for a given application. A producer that sends `acks=1` gets a `PolicyViolationException` naming the problem: *invalid value for 'acks': 1, valid value is one of -1*.
- **Rollout.** Scope it to one service account, a virtual cluster, or everything. Start with `action: INFO`, which logs every producer that would have been rejected without rejecting any, then switch to `BLOCK`.

```yaml
apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: guard-on-produce
  scope:
    username: payments-app
spec:
  pluginClass: io.conduktor.gateway.interceptor.safeguard.ProducePolicyPlugin
  priority: 100
  config:
    acks:
      value: [-1]
      action: BLOCK
    compressions:
      value: ["ZSTD", "LZ4"]
      action: BLOCK
```

### A consumer group that rebalances on every restart

Consumers in the same group divide a topic's partitions between them, and whenever a member joins or leaves, the group pauses and redistributes. That pause is a rebalance. By default a consumer that restarts counts as a new member, so a rolling deploy of six consumers is six rebalances, and during each one nothing in the group is reading.

Kafka's fix is static membership: a consumer that sets `group.instance.id` to a stable name is recognised when it comes back, and the group holds its partitions for it as long as it returns within `session.timeout.ms` (45 seconds by default). It's been in Kafka since 2.3, and it's off unless the application sets it.

- **The mistake.** No `group.instance.id`, so every deploy is a full rebalance.
- **Why the broker can't catch it.** A `JoinGroup` request either carries a `group.instance.id` or it doesn't, and the broker accepts both.
- **What the policy does.** Accepts only the first kind. A `JoinGroup` without one is rejected.
- **Caveat.** The `JoinGroup` response has no field for an error message, so the consumer sees a generic failure and the reason appears in Gateway's audit log. Run it in `INFO` mode first and that audit log is a list of every consumer in the fleet not using static membership, which is usually the most useful output of the whole exercise.

```yaml
apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: enforce-static-group-membership
spec:
  pluginClass: io.conduktor.gateway.interceptor.safeguard.ConsumerGroupPolicyPlugin
  priority: 100
  config:
    groupInstanceId:
      value: .*
      action: BLOCK
```

### A deploy that never stops rebalancing

Every consumer group has one broker acting as its coordinator, and every rebalance runs through it. When a deployment is crash-looping or a large group is rolled quickly, rebalances arrive faster than the coordinator can finish them, which is the pile-up in the timeline above.

- **The mistake.** A fast or unstable deploy that rejoins the group faster than it can settle.
- **Why the broker can't catch it.** Its request quota slows a client that uses too much CPU, but it can't tell a `JoinGroup` from a fetch, so it can't target rebalances.
- **What the policy does.** `LimitJoinGroupPolicyPlugin` caps `JoinGroup` requests per minute, with a separate limit for each consumer group, so one team's crash loop can't spend another team's allowance. Size it to the group: under the classic protocol every membership change makes every member rejoin, so 50 to 100 a minute suits 10 to 20 consumers.
- **Caveat.** It watches the classic protocol's `JoinGroup`. A consumer on the KIP-848 protocol sends `ConsumerGroupHeartbeat` instead, which it doesn't see.

### A consumer that commits after every record

After a consumer processes records, it commits its position so a restart picks up where it left off. By default that happens every five seconds, about seventy commits a minute for a group of six. Some applications turn auto-commit off and commit after every record so that a crash never replays more than one, and a consumer handling a few hundred records a second is then sending a few hundred commits a second.

- **The mistake.** `enable.auto.commit=false` and a commit inside the poll loop.
- **Why the broker can't catch it.** Every commit is a write to `__consumer_offsets` on the coordinator broker, the same one handling the group's rebalances, and the broker has no per-group limit on them.
- **What the policy does.** `LimitCommitOffsetPolicyPlugin` caps `OffsetCommit` requests per minute, per consumer group. It never rejects a commit; it throttles, and the wait grows with how far over the limit the client is. A slightly chatty consumer barely notices, a runaway one is slowed hard. Twenty a minute leaves a healthy small group untouched.

Together, the two limits change what the coordinator spends its minute on:

## Where to Start

1. **Turn on the broker's limits.** `max.connections.creation.rate` and a request quota. They protect the broker from load; connection-rate limits have been there since Kafka 2.7 and request quotas far longer.
2. **Fix the client defaults that should have been defaults.** `group.instance.id` on every consumer, `CooperativeStickyAssignor` first, backoff caps well above one second.
3. **Enforce the client settings the broker can't see, at the proxy.** Start every policy in `INFO`, read the audit log for a week, then move to `BLOCK`. The teams you'd have had to convince one by one never change anything.

Adding brokers treats the symptom. Central traffic enforcement is the remedy.

---

Related: [What a Kafka Proxy Can Do: From Routing to Enforcement →](https://www.conduktor.io/blog/what-a-kafka-proxy-can-do) · [Most Kafka Guardrails Don't Protect Your Data →](https://www.conduktor.io/blog/kafka-guardrails-data-protection) · [Quotas and Rate Limiting in Kafka →](https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka)