Kafka Latency: Measuring p99 & p999 Tails

Stéphane Derosiaux January 30, 2026 12 min read
Flat wireframe line-art on dark teal: a histogram curve rises in a tall hump on the left and stretches into a long, flat tail to the right. Small data-packet cubes with dashed trails cluster under the hump and thin out along the tail; the last straggler, far out on the right, glows lime.

"Our Kafka latency is 5 ms" tells you nothing about whether your users are happy.

An average hides the requests people complain about. The slowest 1% can take 100 times longer than the median, and those are the ones that turn into tickets.

Allegro's Kafka team saw exactly that: single-digit-millisecond medians on produce requests, while p99 reached 1 second and p999 3 seconds. Moving brokers to XFS cut the produce requests over their 65 ms SLO by 82%.

A latency number means something only with its percentile (p50, p99 or p999), its two endpoints (producer to broker, or end to end including your processing) and the window it covers. Benchmarking Kafka means reading p99 and p999: the mean says nothing about the tail.

What does this post change for your team?ClaudeChatGPT

What p99 and p999 Mean

A percentile counts records. p99 is the latency 99% of records beat, so one record in 100 is slower, and p999 (also written p99.9) leaves one in 1,000 above it. At a steady rate, that's a fixed number of slow records:

PercentileRecords slower than itAt 5,000 records/s
p501 in 22,500 per second
p991 in 10050 per second
p9991 in 1,0005 per second
A benchmark of 150,000 records at 5,000 per second on a freshly started three-broker cluster (Kafka 4.3.1 in Docker, on a laptop), with --warmup-records 25000, printed two summaries of the same run:
150000 records sent, 4999.333422 records/sec (4.88 MB/sec), 6.75 ms avg latency, 354.00 ms max latency, 3 ms 50th, 5 ms 95th, 229 ms 99th, 300 ms 99.9th.
125000 steady state records sent, 4998.600392 records/sec (4.88 MB/sec), 2.49 ms avg latency, 9.00 ms max latency, 3 ms 50th, 5 ms 95th, 6 ms 99th, 7 ms 99.9th.

With the warm-up records counted, p99 is 229 ms; without them, 6 ms. The median is 3 ms in both.

At Allegro, the slow produce requests were waiting on ext4 journal commits. One slow write could cascade into many slow requests, and the team traced individual requests to find it:

"p99 and p999 analysis is sometimes not enough."

Allegro Tech Blog

Where a Record Spends Time

Between send() and the consumer's code, a record can wait in four places:

producerbatchlinger.ms, batch.sizesend bufferbuffer.memory1leader brokerrequest queuenum.io.threadslog appenddisk, page cache2consumerfetchup to high watermarkyour codebetween poll() calls4producefetchackfollowerfollower3acks=all: the leader acks once every ISR follower has the recordtheir fetches also move the high watermark
With acks=all the producer waits for step 3. The consumer waits for it whatever acks is set to, because it reads only up to the high watermark.

Each wait has its own settings:

  • In the producer. A batch leaves when it reaches batch.size or after linger.ms (5 ms by default). Past what the brokers absorb, batches queue in buffer.memory.
  • On the leader. The request waits for a handler thread, then for the append to the log. The OS flushes to disk, but the append still waits when the file system blocks, as at Allegro.
  • For the followers. With acks=all, the leader answers the producer only after every replica in the in-sync replica set (ISR) has fetched the batch.
  • In the consumer. A consumer reads only up to the high watermark, the offset every in-sync replica has reached. Then your code processes the record before the next poll().

On a healthy cluster, these settings move latency by a few milliseconds. The seconds in p99 and p999 come from one wait that stalled.

How acks Affects Latency

acks=all Waits for the Slowest ISR Follower

The producer documentation defines acks=all plainly: the leader "will wait for the full set of in-sync replicas to acknowledge the record." The topic setting next to it reads like a quorum:

"Specifies the minimum number of in-sync replicas (including the leader) required for a write to succeed when a producer sets acks to "all""

Apache Kafka, topic configs

🚫 "With three replicas and min.insync.replicas=2, acks=all waits for the fastest two."

min.insync.replicas is a floor: once the ISR shrinks below it, the leader refuses acks=all writes with NOT_ENOUGH_REPLICAS. Above it, the leader still waits for every replica in the ISR, so one slow follower holds every write.

On the lab cluster, orders has three replicas and min.insync.replicas=2. One follower paused for five seconds during an acks=all run at 2,000 records per second:

$ docker pause broker-3; sleep 5; docker unpause broker-3
40000 records sent, 1997.403376 records/sec (1.95 MB/sec), 667.89 ms avg latency, 5052.00 ms max latency, 6 ms 50th, 4093 ms 95th, 4865 ms 99th, 5034 ms 99.9th.

The median stayed at 6 ms while p95 reached 4.1 seconds. A 20-second pause stalled writes for 8.1 seconds, until the controller fenced the paused broker after broker.session.timeout.ms (9 seconds by default) and the leader dropped it from the ISR.

A follower that keeps heartbeating but falls behind, on a slow disk for instance, stays in the ISR for up to replica.lag.time.max.ms, 30 seconds by default.

The leader's request histograms split a produce request's time into five stages. After the 20-second pause, their maxima read:

TotalTimeMsRequestQueueLocalRemoteResponseQueueResponseSendhandler threadlog appendfollowers fetchresponse queuesocket writemax 7 msmax 43 msmax 8,105 msmax 36 msmax 3 msacks=all only
Each box is a broker histogram, kafka.network:type=RequestMetrics,name=<box>TimeMs,request=Produce, read over JMX.

The disk append never passed 43 ms; the 8.1 seconds went to waiting for the follower.

acks=1 Hides the Wait From the Producer Only

Latency tuning guides recommend acks=1, and the producer documentation says why: the leader answers after writing to its own log, without waiting for the followers. That holds for the producer.

Take the same five-second pause with acks=1, now with a console consumer timing each record from its CreateTime:

producer: 40000 records sent, 1998.900605 records/sec (1.95 MB/sec), 3.64 ms avg latency, 134.00 ms max latency, 4 ms 50th, 7 ms 95th, 8 ms 99th, 10 ms 99.9th.
consumer: 40000 records consumed, p50 7 ms, p99 4871 ms, p99.9 5052 ms, max 5063 ms

The producer's p99 stayed at 8 ms. The consumer's was 4.9 seconds, the same as with acks=all (4,866 ms), because the paused follower held the high watermark back:

one follower paused, 5 sacks=1acks=allproducer ackconsumer readsproducer ack, consumer readssendsend0 s123456 s
A record sent as the follower stops: acks=1 acknowledges it at once, and consumers still wait for the follower.

For an end-to-end latency target, acks=1 buys nothing while a follower stalls, and it has other effects:

  • Durability. If the leader fails after acknowledging a record "but before the followers have replicated it then the record will be lost."
  • Idempotence. acks=1 turns it off unless you enabled it explicitly.
  • Under-min ISR. Below min.insync.replicas, KIP-966 stops the high watermark: an acks=1 record stayed unreadable until the follower came back.

Keep acks=1 for producers that must not block on a follower and can lose a record, such as telemetry.

Low Latency vs High Throughput: Pick One

The settings that cut latency cost throughput, and the reverse. Each one turns in opposite directions for the two goals:

SettingFor latencyFor throughputWhat it changes
linger.ms05 (default) or moreHow long a batch waits to fill before it leaves
batch.size16384 (default)LargerHow many records one request carries
compression.typenone (default)lz4 or zstdBytes on the network and disk, against CPU on both ends
fetch.min.bytes1 (default)LargerHow much data the broker waits for before it answers a fetch, up to fetch.max.wait.ms (500 ms)
acks isn't one of these dials. As the section above shows, acks=1 speeds up the producer's acknowledgement, and the consumer still waits for every in-sync follower.

Which side a topic belongs on depends on who waits for each record:

  • Payments, fraud checks, order routing. A customer or a downstream service waits for each record: keep batches short and fetches immediate.
  • Clickstream, logs, CDC into a lake. Nobody waits for one event. Records per second and the bytes you pay for decide, so batch and compress.

A topic that needs both needs scale: more partitions and consumers keep each producer and broker below the rate where records start to queue. Past that rate, the unthrottled benchmark below kept each record 489 ms in the producer's send buffer.

Which of our topics should we tune for latency and which for throughput, given who waits for each record?
Kafka Latency: Measuring p99 & p999 TailsClaudeChatGPT

Where p99 Latency Comes From

A p99 spike comes from one wait that stalled. Each of these causes stalls one of them, and the broker's stage histograms show which:

CauseWhat waitsWhat to changeWhere it shows
Full send bufferRecords in buffer.memory, before any requestProduce at a rate the brokers absorb, or add partitions and brokersrecord-queue-time-max, bufferpool-wait-ratio
GC pause on a brokerEvery request on that brokerG1 with a fixed heap: the start script sets 1 GB, the Apache docs' example 6 GBEvery stage at once, and the broker's kafkaServer-gc.log
Busy request handlersRequests, for an I/O threadMore num.io.threads (8 by default), leaders spread across brokersRequestQueueTimeMs, RequestHandlerAvgIdlePercent
File system stallsThe append to the leader's logXFS, as at Allegro, and Kafka data on disks of its ownLocalTimeMs
Slow or paused followeracks=all writes and the high watermarkFix or replace the broker; it leaves the ISR when fenced or after replica.lag.time.max.msRemoteTimeMs, UnderReplicatedPartitions, IsrShrinksPerSec
Saturated networkResponses, to be sentCompression, more num.network.threads (3 by default), faster NICsResponseSendTimeMs, NetworkProcessorAvgIdlePercent
Consumer rebalanceProcessing, while partitions moveStatic membership (group.instance.id) or group.protocol=consumerLag in seconds, rebalance-latency-max
To find the row behind a spike, compare the stage histograms over the same minutes. The stage whose p99 jumped names the wait, and a jump in every stage at once points to a GC pause.

Benchmarking Kafka Latency

A benchmark without a throughput limit measures the producer's queue. Unthrottled, the same producer pushed 58,893 records per second, and its own metrics show where the half second went:

$ kafka-producer-perf-test.sh ... --throughput -1 --print-metrics
500000 records sent, 58892.815077 records/sec (57.51 MB/sec), 490.37 ms avg latency, 548.00 ms max latency, 501 ms 50th, 541 ms 95th, 545 ms 99th, 547 ms 99.9th.
producer-metrics:record-queue-time-avg:{client-id=perf-producer-client}  : 489.139
producer-metrics:request-latency-avg:{client-id=perf-producer-client}    : 1.242

Records spent 489 ms on average in the 32 MiB send buffer and 1.2 ms in a request to the broker. Benchmark at the rate production runs, and change one setting per run:

  • Report the window and repeat runs. Warm-up moved p99 from 6 ms to 229 ms, and identical runs gave a p999 between 7 and 23 ms.
  • Keep one tool across clusters. OpenMessaging Benchmark runs defined workloads and reports publish latency up to p99.9.
  • Measure in your consumers. Record processing time minus the record's CreateTime in a histogram such as HdrHistogram.

Monitor Latency Distributions

A dashboard of averages hides every tail problem. The warm-up run from the first section, summarized the way such a dashboard would:

StatisticValueWhat it tells you
Average6.75 msLooks healthy
p503 msThe typical record
p955 msStill fine
p99229 msOne record in 100 waited 76 times longer than the median
p999300 msOne in 1,000 waited 100 times longer
The producer's built-in metrics give only an average and a maximum per client. The brokers expose 50thPercentile through 999thPercentile over JMX on the RequestMetrics histogram of each stage, the …TimeMs names in the table above.

Alert when p99 stays above your threshold for several minutes, and on the conditions behind the tail: under-replicated and under-min-ISR partitions, and consumer lag in seconds.

A heatmap of the latency histogram over time shows what a single percentile hides. A p99 that spikes on a fixed period points at something that runs on that period, such as a cron job or a batch consumer catching up.

Which of our consumers need a p99 latency alert, and what threshold and window should each one use?
Kafka Latency: Measuring p99 & p999 TailsClaudeChatGPT

What Percentiles Can't Show

Percentiles describe a distribution of records, and some latency problems stay outside it:

  • A stall on a quiet topic. At one record per second, a two-second stall delays 2 of the hour's 3,600 records and the hourly p999 doesn't move. Count records over a threshold, as Allegro did.
  • Several brokers. A cluster's p99 isn't the average of the brokers' p99s: merge the histograms first.
  • Two clocks. Latency from CreateTime subtracts the producer host's clock from the consumer's, so clock skew shows up as latency.
  • Your processing. A consumer slower than its input builds lag, and each record waits longer than the last.
  • Distance. With followers in other availability zones, every acks=all write and every high watermark move includes the round trip between zones.

Latency Alerts and Chaos Tests

Conduktor Console alerts on the three conditions behind the tail (TimeLag per consumer group, UnderReplicatedPartitionCount, UnderMinIsrPartitionCount) in Slack, Teams, email or webhooks. It collects metrics every 30 seconds, so it tracks sustained stalls; the broker histograms catch a five-second one.

Alerts list in Conduktor Console: time lag alerts on two consumer groups and an under-min-ISR partitions alert firing, an under-replicated partitions alert healthy
Lag and ISR alerts in Console.

To test your clients' timeouts against a slow tail, Gateway's slow-broker chaos interceptor delays a share of responses by a random time you bound.

Latency SLOs at the Consumer

The latency a customer sees is the consumer's: the time from a record's CreateTime to the code that acts on it. It includes the slowest in-sync follower's wait and the consumer's lag, whatever acks the producer uses, so a producer's p99 can't replace it.

Two kinds of teams will ask the platform for p99 and p999 at their consumers, at production rate and over a stated window, and for an alert when a follower stalls:

  • Teams with a consumer SLO. A service that ships orders sets a latency SLO on the records it consumes.
  • AI agent rollouts. An agent that answers from the latest events in Kafka is a consumer with a deadline.

Talk to us about latency alerts on your clusters →


Related: Kafka Performance: Measure Before You Tune → · JVM Tuning for Kafka Brokers: G1GC vs ZGC → · Kafka Consumer Lag Alert Thresholds →