Kafka Partitions are the wrong ordering abstraction. Keys are.

Stéphane Derosiaux June 1, 2026 9 min read

Flat wireframe line-art on a dark teal background: a single braided rope arrives from the left, passes through one glowing lime ring, and comes out as three separate parallel strands of evenly spaced beads racing off the right edge

Confluent has officially deprecated the Parallel Consumer library, redirecting people to Share Consumers in Apache Kafka as the replacement, "similar functionality". Chuck, one of our engineers at Conduktor, called this out on LinkedIn this week. I want to second him, loudly: this is the wrong call. Not the fact to deprecate this lib but to say that "Share Consumers" are the replacement. They are a completely different beast. They solve a different problem.

Antony Stubbs, one of the original implementers of Parallel Consumer on Confluent's CSID team, has forked it and is actively improving it, thanks Antony: github.com/astubbs/parallel-consumer.

So, what's the deal here?

I spent years building event-driven and event-sourcing systems before Conduktor. The kind of systems where the partition is a deployment detail and the key (the entity id, whatever your domain calls it) is the entire universe: customer_id, order_id, etc.

Partitions are the wrong ordering abstraction, keys are, and this library was "fixing" Kafka. Parallelism + keys + offset commit correctness is the real deal. It is exactly what event-driven systems and event sourcing need, where the key is all that matters: you want parallelism across keys, and no head-of-line blocking because one key or one external call is slow.

Almost every "we need more parallelism on this consumer" problem in an event-driven system is really a "we're using the wrong abstraction" problem.

Let me walk through it.

What is Parallel Consumer?

Parallel Consumer is a library that wraps the vanilla Kafka consumer and lets a single consumer instance process records in parallel across configurable units of granularity:

Mode	Ordering it guarantees	Parallelism	What it's for
`KEY`	Records sharing a key are processed sequentially in the application	Across keys: different keys run at the same time	Event-driven systems, event sourcing, anything where an entity's sequence matters
`UNORDERED`	None	Maximum. Just crunch through the records in parallel	Independent tasks where order is irrelevant
`PARTITION`	Per-partition, same as the vanilla consumer	One worker thread per assigned partition	Keeping vanilla semantics, without multiplexing N partitions sequentially on one thread

KEY is the default, and it's the one you want.

The challenge it's solving is to parallelize processing in the application while keeping offset commit correctness, as Kafka does not support ack-per-message but only commit-per-batches.

Why is this fixing Kafka?

Because it parallelizes within the partitions already assigned to that consumer instance. It doesn't steal partitions from other members of the group, and it doesn't touch the rebalance protocol at all.

One instance assigned 10 partitions can process each batch in parallel: 10, 100, 1000, whatever you set as maxConcurrency, and the lib will keep ordering everything by key/timestamp.

You decouple your processing concurrency from your partition count.

This is the entire pitch of this lib.

Share Consumers (KIP-932, "Queues for Kafka") are great and production-ready since 4.2 (February 2026). They give Kafka a queue-like consumption model on top of the same topic you used to consume.

Each record is individually acquired by a consumer, then processed, then acknowledged. In terms of actions: ACCEPT, RELEASE (put it back for retry), or REJECT (dead-letter it).

The record lifecycle becomes: Available → Acquired → Acknowledged.

It's per-message acks.

The big difference with parallel-consumer: there is no per-partition order and no per-key order. The KIP: "The records in a share-partition can be delivered out of order to a consumer." Can = Will.

For job-queue use cases (process this PDF, send this email, run this image transformation), Share Consumers are excellent. Each record is an independent task, order doesn't matter, you want max throughput and per-record retry.
For event-driven systems, event sourcing? Ordering per key is the entire point.
For AI agents, it depends on their logic and context needs.

If UserUpdated(user=42) arrives at offset 10 and UserDeleted(user=42) arrives at offset 50, you'd better process them in that order. Otherwise you've just resurrected a deleted user.

Telling people to migrate from Parallel Consumer to Share Consumers is like telling people to migrate from PostgreSQL to Redis because both store data. Sure, but the semantics and the behavior are vastly different.

Partitions are an infrastructure concern. Keys are the domain concern.

The partition is a storage and assignment unit. It exists because Kafka needs a way to shard data across brokers and distribute consumption across instances. It's a side effect of horizontal scaling. The partition count is a deployment parameter: you pick a number, hope it's enough, and live with the consequences.

The key, on the other hand, is what your domain actually cares about. user_id, order_id, account_id. The business contract is: "all events about the same entity must be processed in order". Nobody in the business cares whether user_42 lives on partition 7 or partition 23. They care that the sequence of events on user_42 is respected.

The partition is the wrong abstraction for ordering, it's a leaky one that couples your domain's ordering guarantee to a deployment constraint. If you bump partitions to chase throughput, you'll enjoy reporting processing errors and correctness issues downstream.

And the partition count is a one-way door: Kafka never lets you bring it back down. So the usual reflex, adding partitions until the consumer keeps up, turns a processing problem into permanent infrastructure. More partitions to replicate, longer leader elections when a broker dies, more work for the controller, and a bigger bill on every cloud service that prices per partition.

That's how you end up with over-partitioned topics whose consumer group only ever runs three members. The partitions were never what you needed. The parallelism was.

Another consequence of this partition model is the head-of-line blocking issue: slow records block other records that have no reason to be blocked. If processing user_42 takes 4 seconds, all records following on the same partition will have to wait 4 seconds, just because they're behind user_42 in line.

One slow key blocks unrelated keys on the same partition: key A takes 4s while keys B, C and D, each 100ms, wait behind it

The vanilla consumer model is: one partition = one unit of work = one thread. It falls apart the moment your processing is IO-bound: external HTTP call, database write, ML inference, external enrichment, RAG retrieval, LLM call (yep), anything synchronous. And let me tell you, that's VERY common.

So the question becomes:

How do you process records in parallel across keys, without breaking per-key ordering, without leaking ordering bugs into your domain, and without committing offsets wrong?

That's exactly what Parallel Consumer was built for.

The hard part: offset commit correctness

Clearly underestimated. Parallel processing across keys is the easy part, just fan out by key into a thread pool, right? Not so fast.

Imagine you're processing partition 7. The records come in at offsets 100, 101, 102, 103, ..., 200. You're processing them in parallel, by key.

Record at offset 105 (key A) takes 4 seconds because it's slow.
Records 106, 107, 108 (keys B, C, D) finish in 100ms each.

What offset do you commit?

Offsets 100 to 104 done, 105 still processing after 4s, 106 to 108 done but not yet committable: the highest contiguous offset is 104

Two wrong answers:

If you commit 108 and if the consumer crashes, you'll lose record 105 on restart.
If you wait for 105 to finish before committing anything, you've reintroduced head-of-line blocking.

The right (complicated) answer:

You keep an in-memory map of completed offsets per partition, you commit only the highest contiguous offset that has been fully processed, and you keep the "gaps" (completed but blocked-by-earlier-incomplete records) in memory so the system can recover correctly on restart.

This is non-trivial, and you need to think about memory, backpressure, rebalancing situation, key fairness. Parallel Consumer gets all of this right. It stuffed this state into the offset commit metadata string in Kafka (that nobody ever used).

"Just use virtual threads"

Java 21 brought virtual threads, and people have been (rightly) excited about them for IO-bound workloads. You can have millions of them, and they don't block OS threads on IO. So you can spin up a virtual thread per partition or per key. Well...

Virtual threads solve the concurrency cost problem, and avoid blocking an OS thread on a synchronous HTTP call. They do not solve the ordering and offset-commit problem on their own.

Two layers: the contract layer (ordering by key, offset commit, backpressure, rebalance) handled by Parallel Consumer, and the execution layer of virtual threads. Different concerns

Spawning a virtual thread per partition barely improved on the vanilla consumer, it would achieve the same as with a regular thread pool. If you spawn a virtual thread per key on the other hand, you need:

A way to track which keys are in flight (so you don't dispatch two threads for the same key)
A completed-offset tracker, including that 4 KB metadata-encoding dance from the previous section, or your restarts reprocess or lose records
Backpressure
Rebalance handling
Bounded memory

It's exactly what Parallel Consumer is doing. There's a new library called kpipe (github.com/eschizoid/kpipe) that wraps virtual threads on top of the vanilla Kafka consumer and gives you some of this. I haven't run it yet.

TLDR: This should be in Apache Kafka

The Kafka consumer should ship with key parallelism out of the box. Partition parallelism is a terrible default when you work with event-driven architecture and AI agent payloads. There should be a KIP for first-class parallel processing in org.apache.kafka.clients.consumer.

If you're wondering whether someone already proposed this: sort of, and the story is telling. This is not a new need, I discovered there is even an old KIP-attempt from 2019, "KIP-X: a cooperative consumer processing semantic".

So what should I do tomorrow?

If you're running event-driven workloads with synchronous processing (billing, payments, enrichment, anything where a single record takes more than a few milliseconds) and you're not using Parallel Consumer or an equivalent, you're almost certainly wasting partitions or time.

Remember this:

One question decides it: does per-key order matter?

Another path: Client-specific virtual partitioning

What if the platform team could help without changing the application code? With a Kafka proxy, a slow consumer does not have to see Kafka exactly as it is.

For one specific app, the proxy could expose more virtual partitions than the topic really has, then map them back to the real partitions behind the scenes. The app gets more horizontal scaling room. The platform team keeps control. No library migration. No app rewrite.

When you sit in the protocol path, you can bend the Kafka protocol to add the missing abstractions Kafka never gave you.