Kafka Queues: Share Groups Explained

Stéphane Derosiaux July 18, 2026 10 min read

Kafka share groups are a consumption model introduced in KIP-932 that lets many consumers cooperatively read one topic with per-record acknowledgement, so multiple members can process records from the same partition at once.

This solves a long-standing Kafka limitation by decoupling consumer parallelism from partition count, bringing queue-like semantics to Apache Kafka without changing the underlying partitioned log.

A classic consumer group binds each partition to exactly one member at a time. Concurrency is therefore capped at the partition count: ten consumers on a six-partition topic leave four idle. Raising throughput has historically meant adding partitions, which can have negative consequences for the cluster.

Share groups remove that coupling. A share group is a named set of consumers/workers where multiple members can process records from the same partition concurrently, and each record is acknowledged individually rather than by advancing a partition offset. The broker distributes records, tracks their state, and reclaims any that are not acknowledged in time to redistribute them to other workers. This effectively gives a shared work queue on top of the Kafka log, with two consequences:

Parallelism is no longer tied to partition count. A share group can have more active members than the topic has partitions.
Failed records can be redelivered or discarded per record, without stalling the rest of the stream (avoiding head-of-line blocking).

Dimension	Classic consumer group	Share group
Partition sharing	One consumer per partition	Many consumers per partition
Parallelism ceiling	Number of partitions	Independent of partition count
Acknowledgement	Offset commit (high-water mark)	Per record: accept / release / reject
Ordering	Guaranteed within a partition	Within a delivered batch only; none across batches or members
Redelivery	Rewind offset (whole group)	Per record, up to a delivery-attempt limit
Assignor	Client- or server-driven	Share-partition assignor (server-driven)
Best fit	Ordered streams, high throughput	Job/task queues, slow I/O, AI agents ⭐

What to remember: share groups exchange strict per-partition ordering for the ability to spread work across arbitrarily many consumers and scale horizontally.

How per-message acknowledgement works

Instead of committing an offset, a share consumer resolves each record individually. Three outcomes:

ACCEPT: the record was processed successfully; it's marked acknowledged and will never be redelivered.
RELEASE: the consumer cannot process it right now (a transient failure); the record is redelivered to another consumer.
REJECT: the record is unprocessable (e.g. an unexpected payload or a critical error); it is archived with no further attempts.

A consumer that needs more time (slow I/O, high latency, LLM calls...) can also RENEW the acquisition lock to extend processing without releasing the record.

Each record moves through a small state machine per share group: Available → Acquired → Acknowledged (ACCEPT), or back to Available (RELEASE), or to Archived (REJECT, or once retries are exhausted).

When a consumer acquires a record it holds an acquisition lock for share.record.lock.duration.ms (default 30s). If the record is not acknowledged before the lock expires (a crash, a hang, a slow downstream call) it automatically returns to Available for redelivery.

There is also built-in protection against poison-pill messages. Every acquisition increments a delivery count; when it reaches group.share.delivery.count.limit (default 5) the record is archived and no longer delivered, so a single bad record cannot loop forever.

Share groups help most when the bottleneck is the consumer itself:

I/O-bound work. Each record triggers a slow external call: an HTTP request, a database write, an LLM inference. Consumers spend most of their time waiting, so adding more of them raises effective throughput even on a single partition.
More consumers than partitions. You want 50 workers but the topic has 6 partitions. A share group admits all 50 without repartitioning.
Head-of-line blocking. In a classic group, one slow record blocks its partition. With per-record acquisition, other members keep draining while the slow one is retried or archived.
Job and task queues. Workloads that would otherwise force over-partitioning purely to raise concurrency map cleanly onto a share group.
AI agents and multi-agent systems. A single agent step is an LLM call or tool invocation measured in seconds, not milliseconds, so a stream of agent tasks is dominated by consumer wait time rather than broker throughput. When an orchestrator fans work out to specialized sub-agents over Kafka, a share group runs far more agent workers than the topic has partitions, and per-record acknowledgement retries a slow or failed step (an inference timeout, a provider rate limit) without stalling the rest. That makes Kafka a natural backbone for multi-agent workloads, where the bottleneck is model latency, not the log.

Ordered per-key processing. CDC and any state machine that depends on strict in-partition order need a classic consumer group. Share groups give no ordering across batches, members, or redeliveries.
Event-driven architecture and event sourcing. Ordering is critical: entity events must be processed in order (create, then update) to form a deterministic state. A share group instead splits records among its members like a work queue and acknowledges them one at a time, so no single member sees the whole ordered stream.
Raw maximum throughput. For firehose pipelines where consumers are already CPU- or network-saturated, classic consumer groups with large fetch batches and sequential I/O win. Per-record acknowledgement and lock tracking add overhead that pure throughput workloads don't want to pay.
Exactly-once transactional chains. The mature transactional/EOS tooling is built around offset commits, not share-group acknowledgement.

Share groups have followed a staged rollout:

Kafka 4.0: Early Access, test-only, not upgradeable, wire format not stable.
Kafka 4.1: Preview (not wire-compatible with the 4.0 early access).
Kafka 4.2: production-ready / GA.

kafka-features.sh --bootstrap-server localhost:9092 describe | grep share
# Feature: share.version  SupportedMinVersion: 0  SupportedMaxVersion: 1  FinalizedVersionLevel: 1

The share rebalance protocol has to be turned on: group.coordinator.rebalance.protocols must list share. This applies to every cluster, single-broker or not. Without it, share consumers join but receive empty assignments (and therefore zero records):

group.coordinator.rebalance.protocols=classic,consumer,share

Share-group state (locks, delivery counts, start offset) lives in a new internal topic __share_group_state, managed by a share coordinator. That topic carries its own replication settings:

share.coordinator.state.topic.replication.factor=3 # Set to 1 if single broker in dev
share.coordinator.state.topic.min.isr=2 # Set to 1 if single broker in dev

There is a new Java API: class KafkaShareConsumer (not your classic KafkaConsumer) and the CLI adds kafka-console-share-consumer.sh for consuming and kafka-share-groups.sh (--list, --describe, --members, --state) for administration.

Kafka queues vs MQ alternatives

Share groups add queue semantics, but they are not a drop-in RabbitMQ or ActiveMQ Artemis replacement. Kafka still has many gaps:

No built-in dead-letter queue.
No delayed or backoff retry.
Log-retention model, not queue-drain.

So share groups are best understood as queue-like consumption over a durable log, closing the biggest gap that pushed teams toward a separate broker (see Kafka vs RabbitMQ) without adopting RabbitMQ's full feature surface. Tools like Conduktor sit alongside the native CLIs to give operators visibility into share-group state, member assignments, and redelivery behavior across environments.

Do share groups replace consumer groups?

No. They are an additional consumption model for queue-like, unordered work. Ordered per-key streams, high-throughput pipelines, and exactly-once chains still use classic consumer groups.

Can more consumers than partitions be active in a share group?

Yes. That is the point: parallelism is decoupled from partition count, so a share group can run more members than the topic has partitions, all sharing the same partitions.

Are share groups production-ready?

They are GA in Apache Kafka 4.2, gated by the share.version feature flag (finalized to level 1 by default). They were Early Access in 4.0 and Preview in 4.1.

What happens to a record that keeps failing?

Each delivery increments a count; once it hits the limit (default 5, group.share.delivery.count.limit) the record is archived and never redelivered. There is no built-in dead-letter queue yet.

Do share groups guarantee ordering?

Only within a single delivered batch for a share-partition. There is no ordering guarantee across batches, across members, or across redeliveries.

Is Kafka a message queue?

Not by default: a topic is a retained, replayable log, and records are not deleted on read. Share groups (KIP-932) add queue-like consumption on top (per-record acknowledgement, many consumers per partition), so Kafka can serve queue workloads while staying a log. See Kafka vs RabbitMQ.

What is the difference between a Kafka topic and a queue?

A topic is a durable log that many consumer groups each read in full and can replay; a traditional queue distributes each message to one worker and drops it on ack. A share group adds queue-style distribution over a topic, but the log underneath stays retained and replayable.

Kafka Consumer Groups Explained: the classic one-consumer-per-partition model share groups extend.
Kafka Partitions Explained: why partition count historically capped consumer parallelism.
Kafka Topics Explained: the durable log that share groups consume from.
Kafka vs RabbitMQ: how log-based streaming compares to a traditional message queue.
Dead Letter Queues for Error Handling: the DLQ pattern share groups still leave to the application.