# Kafka Queues: Share Groups Explained

**Kafka share groups** are a consumption model introduced in [KIP-932](https://kafka-options-explorer.conduktor.io/kip/932/) that lets many consumers cooperatively read one topic with per-record acknowledgement, so multiple members can process records from the same partition at once.

This solves a long-standing Kafka limitation by decoupling consumer parallelism from partition count, bringing queue-like semantics to Apache Kafka without changing the underlying partitioned log.

![kafka-share-groups diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-share-groups-0.webp)

## What are Kafka share groups?

A classic [consumer group](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) binds each partition to exactly one member at a time. Concurrency is therefore capped at the partition count: ten consumers on a six-partition topic leave four idle. Raising throughput has historically meant adding partitions, which can have negative consequences for the cluster.

Share groups remove that coupling. A share group is a named set of consumers/workers where **multiple members can process records from the same partition concurrently**, and each record is acknowledged individually rather than by advancing a partition offset. The broker distributes records, tracks their state, and reclaims any that are not acknowledged in time to redistribute them to other workers. This effectively gives a shared work queue on top of the Kafka log, with two consequences:

- Parallelism is no longer tied to partition count. A share group can have more active members than the topic has partitions.
- Failed records can be redelivered or discarded per record, without stalling the rest of the stream (avoiding head-of-line blocking).

## Share groups vs consumer groups

| Dimension | Classic consumer group | Share group |
|---|---|---|
| Partition sharing | One consumer per partition | Many consumers per partition |
| Parallelism ceiling | Number of partitions | Independent of partition count |
| Acknowledgement | Offset commit (high-water mark) | Per record: accept / release / reject |
| Ordering | Guaranteed within a partition | Within a delivered batch only; none across batches or members |
| Redelivery | Rewind offset (whole group) | Per record, up to a delivery-attempt limit |
| Assignor | Client- or server-driven | Share-partition assignor (server-driven) |
| Best fit | Ordered streams, high throughput | Job/task queues, slow I/O, AI agents ⭐ |

What to remember: share groups exchange strict per-partition ordering for the ability to spread work across arbitrarily many consumers and scale horizontally.

## How per-message acknowledgement works

Instead of committing an offset, a share consumer resolves each record individually. Three outcomes:

- **ACCEPT**: the record was processed successfully; it's marked acknowledged and will never be redelivered.
- **RELEASE**: the consumer cannot process it right now (a transient failure); the record is redelivered to another consumer.
- **REJECT**: the record is unprocessable (e.g. an unexpected payload or a critical error); it is archived with no further attempts.

A consumer that needs more time (slow I/O, high latency, LLM calls...) can also RENEW the acquisition lock to extend processing without releasing the record.

Each record moves through a small state machine per share group: `Available → Acquired → Acknowledged` (ACCEPT), or back to `Available` (RELEASE), or to `Archived` (REJECT, or once retries are exhausted).

![kafka-share-groups diagram 2](https://www.conduktor.io/assets/images/glossary/kafka-share-groups-1.webp)

When a consumer acquires a record it holds an **acquisition lock** for [`share.record.lock.duration.ms`](https://kafka-options-explorer.conduktor.io/configuration/?search=share.record.lock.duration.ms#broker) (default 30s). If the record is not acknowledged before the lock expires (a crash, a hang, a slow downstream call) it automatically returns to `Available` for redelivery.

There is also built-in protection against poison-pill messages. Every acquisition increments a delivery count; when it reaches [`group.share.delivery.count.limit`](https://kafka-options-explorer.conduktor.io/configuration/?search=group.share.delivery.count.limit#broker) (default 5) the record is archived and no longer delivered, so a single bad record cannot loop forever.

## When share groups help

Share groups help most when the bottleneck is the *consumer* itself:

- **I/O-bound work.** Each record triggers a slow external call: an HTTP request, a database write, an LLM inference. Consumers spend most of their time waiting, so adding more of them raises effective throughput even on a single partition.
- **More consumers than partitions.** You want 50 workers but the topic has 6 partitions. A share group admits all 50 without repartitioning.
- **Head-of-line blocking.** In a classic group, one slow record blocks its partition. With per-record acquisition, other members keep draining while the slow one is retried or archived.
- **Job and task queues.** Workloads that would otherwise force over-partitioning purely to raise concurrency map cleanly onto a share group.
- **AI agents and multi-agent systems.** A single agent step is an LLM call or tool invocation measured in seconds, not milliseconds, so a stream of agent tasks is dominated by consumer wait time rather than broker throughput. When an orchestrator fans work out to specialized sub-agents over Kafka, a share group runs far more agent workers than the topic has partitions, and per-record acknowledgement retries a slow or failed step (an inference timeout, a provider rate limit) without stalling the rest. That makes Kafka a natural backbone for multi-agent workloads, where the bottleneck is model latency, not the log.

## When share groups do not help

- **Ordered per-key processing.** [CDC](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) and any state machine that depends on strict in-partition order need a classic consumer group. Share groups give no ordering across batches, members, or redeliveries.
- **[Event-driven architecture](https://www.conduktor.io/glossary/event-driven-architecture) and [event sourcing](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka).** Ordering is critical: entity events must be processed in order (create, then update) to form a deterministic state. A share group instead splits records among its members like a work queue and acknowledges them one at a time, so no single member sees the whole ordered stream.
- **Raw maximum throughput.** For firehose pipelines where consumers are already CPU- or network-saturated, classic consumer groups with large fetch batches and sequential I/O win. Per-record acknowledgement and lock tracking add overhead that pure throughput workloads don't want to pay.
- **Exactly-once transactional chains.** The mature transactional/EOS tooling is built around offset commits, not share-group acknowledgement.

## Status, versions, and enabling share groups

Share groups have followed a staged rollout:

- **Kafka 4.0**: Early Access, test-only, not upgradeable, wire format not stable.
- **Kafka 4.1**: Preview (not wire-compatible with the 4.0 early access).
- **Kafka 4.2**: production-ready / GA.

```bash
kafka-features.sh --bootstrap-server localhost:9092 describe | grep share
# Feature: share.version  SupportedMinVersion: 0  SupportedMaxVersion: 1  FinalizedVersionLevel: 1
```

The share rebalance protocol has to be turned on: [`group.coordinator.rebalance.protocols`](https://kafka-options-explorer.conduktor.io/configuration/?search=group.coordinator.rebalance.protocols#broker) must list `share`. This applies to every cluster, single-broker or not. Without it, share consumers join but receive empty assignments (and therefore zero records):

```properties
group.coordinator.rebalance.protocols=classic,consumer,share
```

Share-group state (locks, delivery counts, start offset) lives in a new internal topic `__share_group_state`, managed by a share coordinator. That topic carries its own **replication settings**:

```properties
share.coordinator.state.topic.replication.factor=3 # Set to 1 if single broker in dev
share.coordinator.state.topic.min.isr=2 # Set to 1 if single broker in dev
```

There is a new Java API: class `KafkaShareConsumer` (not your classic `KafkaConsumer`) and the CLI adds `kafka-console-share-consumer.sh` for consuming and `kafka-share-groups.sh` (`--list`, `--describe`, `--members`, `--state`) for administration.

## Kafka queues vs MQ alternatives

Share groups add queue semantics, but they are **not a drop-in RabbitMQ or ActiveMQ Artemis replacement**. Kafka still has many gaps:

- **No built-in dead-letter queue.**
- **No delayed or backoff retry.**
- **Log-retention model, not queue-drain.**

So share groups are best understood as *queue-like consumption over a durable log*, closing the biggest gap that pushed teams toward a separate broker (see [Kafka vs RabbitMQ](https://www.conduktor.io/glossary/kafka-vs-rabbitmq)) without adopting RabbitMQ's full feature surface. Tools like Conduktor sit alongside the native CLIs to give operators visibility into share-group state, member assignments, and redelivery behavior across environments.

**Do share groups replace consumer groups?**

No. They are an additional consumption model for queue-like, unordered work. Ordered per-key streams, high-throughput pipelines, and exactly-once chains still use classic consumer groups.

**Can more consumers than partitions be active in a share group?**

Yes. That is the point: parallelism is decoupled from partition count, so a share group can run more members than the topic has partitions, all sharing the same partitions.

**Are share groups production-ready?**

They are GA in Apache Kafka 4.2, gated by the share.version feature flag (finalized to level 1 by default). They were Early Access in 4.0 and Preview in 4.1.

**What happens to a record that keeps failing?**

Each delivery increments a count; once it hits the limit (default 5, group.share.delivery.count.limit) the record is archived and never redelivered. There is no built-in dead-letter queue yet.

**Do share groups guarantee ordering?**

Only within a single delivered batch for a share-partition. There is no ordering guarantee across batches, across members, or across redeliveries.

**Is Kafka a message queue?**

Not by default: a topic is a retained, replayable log, and records are not deleted on read. Share groups (KIP-932) add queue-like consumption on top (per-record acknowledgement, many consumers per partition), so Kafka can serve queue workloads while staying a log. See [Kafka vs RabbitMQ](https://www.conduktor.io/glossary/kafka-vs-rabbitmq).

**What is the difference between a Kafka topic and a queue?**

A topic is a durable log that many consumer groups each read in full and can replay; a traditional queue distributes each message to one worker and drops it on ack. A share group adds queue-style distribution over a topic, but the log underneath stays retained and replayable.

## Related Pages

- [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained): the classic one-consumer-per-partition model share groups extend.
- [Kafka Partitions Explained](https://www.conduktor.io/glossary/kafka-partitions-explained): why partition count historically capped consumer parallelism.
- [Kafka Topics Explained](https://www.conduktor.io/glossary/kafka-topics-explained): the durable log that share groups consume from.
- [Kafka vs RabbitMQ](https://www.conduktor.io/glossary/kafka-vs-rabbitmq): how log-based streaming compares to a traditional message queue.
- [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling): the DLQ pattern share groups still leave to the application.

## Sources

- [KIP-932: Queues for Kafka (Apache Kafka wiki)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka)
- [Queues for Kafka (KIP-932), Preview Release Notes](https://cwiki.apache.org/confluence/display/KAFKA/Queues+for+Kafka+(KIP-932)+-+Preview+Release+Notes)
- [Apache Kafka 4.2 Group Configs (share.auto.offset.reset, share.isolation.level)](https://kafka.apache.org/42/configuration/group-configs/)
- [KafkaShareConsumer API (Apache Kafka Javadoc)](https://kafka.apache.org/42/javadoc/org/apache/kafka/clients/consumer/KafkaShareConsumer.html)
- [Let's Take a Look at KIP-932: Queues for Kafka! (Gunnar Morling)](https://www.morling.dev/blog/kip-932-queues-for-kafka/)
- [Configure Share Groups for Docker in Confluent Platform](https://docs.confluent.io/platform/current/installation/docker/operations/kafka-queues-docker.html)