# Kafka Isn't a Queue: Stop Designing It Like One

How does a message leave RabbitMQ or SQS? A worker grabs it, acks it, the broker deletes it. Gone. That lifecycle is so universal that most of us carry it into Kafka without noticing, and then design topics around a deletion that never happens.

It shows up the same way in cluster after cluster: single-partition topics "for ordering", aggressive retention "to clean up after ourselves", consumer lag treated as an incident. None of these are Kafka problems. They are queue habits applied to a log.

> **Update, June 2026.** Kafka now ships queue semantics: share groups ([KIP-932](https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka)) went early access in 4.0, preview in 4.1, and are production-ready since [Apache Kafka 4.2](https://kafka.apache.org/blog/2026/02/17/apache-kafka-4.2.0-release-announcement/) (February 2026). Share groups change how you *read* the log, not what Kafka *is*, so the mental model below still decides your design. Covered in detail near the end.

## The queue mental model

A traditional queue works like a mailbox:

1. Producer drops a message
2. A consumer picks it up
3. The message gets deleted
4. The next consumer gets the next message

Multiple workers race to grab messages from a shared pool. Consumption is destructive: once processed, the message no longer exists. The broker tracks who has what, retries what failed, and the backlog physically shrinks as you work.

> 🚫 *"Kafka is basically a faster RabbitMQ."*

Kafka doesn't work this way.

## Kafka is a log

Kafka is an append-only, immutable log:

![Diagram contrasting a queue and a Kafka log. Left: a queue where two consumed messages are gone and a worker's ack deletes the message it holds. Right: a Kafka log where all eight records remain and two readers point at offsets 1 and 4.](https://www.conduktor.io/assets/images/blog/kafka-isnt-a-queue-queue-vs-log.svg)

Records are never deleted by consumers. They stay until retention expires (a storage policy, nothing to do with consumption). Each consumer tracks its own offset. Two consumers can read the same partition at the same time, at completely different positions, and neither affects the other.

**Consuming changes nothing: the log stays, readers move.**

## What the log changes

### Consuming deletes nothing

A bug in your processor? Fix it, reset offsets, rerun:

```bash
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-processor --topic events \
  --reset-offsets --to-earliest --execute
```

This command makes no sense in a queue world (replay *what*? the messages are gone). In Kafka it's a standard recovery move, and it's the foundation of [event sourcing patterns](https://www.conduktor.io/blog/event-sourcing-kafka-patterns-pitfalls): the log is the source of truth, downstream state is derived and rebuildable.

### Ordering is per-partition

Kafka guarantees ordering only within a partition. You can have total ordering or parallelism, not both:

| Approach | Ordering | Parallelism |
|----------|----------|-------------|
| Single partition | Total order | 1 consumer max |
| Key-based partitioning | Per-key order | Up to N consumers |

Key-based partitioning is the sweet spot: all events for a given key (user ID, order ID) land on the same partition, in order, while other keys flow in parallel.

And design against keys, not partition numbers. The key names which entity must stay ordered, a business rule. The partition number is an infrastructure detail, and treating it as your ordering abstraction is a leak. For event-driven and event-sourced systems, [I made the longer argument here](https://sderosiaux.medium.com/kafka-partitions-are-the-wrong-ordering-abstraction-keys-are-b54dc5b20b95).

### Consumer groups divide work

In RabbitMQ, consumers compete for messages. In a Kafka consumer group, they split partitions:

```text
Partition 0 → Consumer 1 (all records from P0)
Partition 1 → Consumer 2 (all records from P1)
Partition 2 → Consumer 3 (all records from P2)
```

Each partition belongs to exactly one consumer in the group. Add a fourth consumer to this topic and it sits idle.

➡ Partition count caps your parallelism. This is the constraint that pushed teams to over-partition for years, and the one share groups finally remove (more on this later).

### Several groups, one topic

The part queues can't match:

![One user-events topic of ten records with three consumer groups pointing at their own offsets: email-sender at offset 2, fraud-detector at offset 6, analytics caught up at offset 9.](https://www.conduktor.io/assets/images/blog/kafka-isnt-a-queue-one-topic-three-groups.svg)

Each group reads every record, independently, at its own pace. No fan-out exchange to configure, no data duplication, no upstream change when a fourth team shows up. You can [monitor all your consumer groups](https://docs.conduktor.io/guide/monitor-brokers-apps) in one place to keep track of who reads what.

## The anti-patterns

Queue thinking produces the same four mistakes:

- **Single partition "for ordering".** You get total order, a hard ceiling of one consumer, and a painful recovery story. Key-based partitioning gives you the ordering you need (per key) without the ceiling.
- **Short retention "to clean up".** Retention is a storage policy, not a consumption policy. Nothing gets "cleaner" because consumers are done with the data. A week is common; a year isn't unusual for audit topics.
- **Treating lag as a bug.** Lag is information. 5,000 records behind could be a slow consumer (problem), an hourly batch job (expected), or a new consumer catching up on history (temporary). Alert on lag *growth rate*, not absolute lag.
- **One topic per consumer.** That's queue-world isolation. It duplicates data and couples producers to consumers. One topic, several consumer groups.

So if the log model is this good, why did the Kafka project just ship queues?

## Share groups: queue semantics on the log (Kafka 4.2)

Because some workloads are queues. A pool of workers chewing through independent jobs (payment captures, image renders, emails) doesn't care about ordering. It cares about spreading records across as many workers as possible and retrying the ones that fail. For years the honest answer was "run SQS or RabbitMQ next to Kafka" (go figure), or over-partition so more consumers could attach.

[KIP-932](https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka), "Queues for Kafka", is the project's answer. Share groups are a second way to consume a topic, production-ready since Kafka 4.2:

- No exclusive partition assignment: the broker hands records to whichever consumer asks
  ➡ consumers in a share group can outnumber partitions, so you stop over-partitioning just to add workers.
- Per-record acknowledgement: `ACCEPT` it, `RELEASE` it for redelivery (transient failure), or `REJECT` it (unprocessable)
- Fetched records are locked while you process them, 30 seconds by default
  ➡ no ack before the lock expires? The record goes back to the pool, delivery count +1. Kafka 4.2 added a way to extend the lock (`RENEW`) for slow records.
- Delivery attempts are capped, 5 by default
  ➡ past the cap the record is archived and skipped, not parked: there is no built-in [dead letter queue](https://www.conduktor.io/blog/dead-letter-topics-handling-poison-pills) yet (KIP-1191 is the proposal).
- Delivery is at-least-once, ordering across a worker pool is gone, and [exactly-once semantics](https://www.conduktor.io/blog/exactly-once-semantics-when-it-works) are not supported.

![A topic with one partition dispatching individual records to three workers in a share group: worker 1 accepts its record, worker 2 releases it for redelivery, worker 3 rejects it. The record held by worker 1 is locked while it is processed.](https://www.conduktor.io/assets/images/blog/kafka-isnt-a-queue-share-group-dispatch.svg)

In code, it's a `KafkaShareConsumer` and acks per record instead of offset commits:

```java
// consumer config: share.acknowledgement.mode=explicit
KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props);
consumer.subscribe(List.of("payment-jobs"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(200));
    for (ConsumerRecord<String, String> record : records) {
        try {
            process(record);
            consumer.acknowledge(record, AcknowledgeType.ACCEPT);
        } catch (RetriableException e) {
            consumer.acknowledge(record, AcknowledgeType.RELEASE); // back to the pool
        } catch (Exception e) {
            consumer.acknowledge(record, AcknowledgeType.REJECT);  // unprocessable
        }
    }
    consumer.commitSync(); // acknowledgements are sent here
}
```

Does that make Kafka a queue now? The release notes themselves are careful about this:

> *"It does not add the concept of a 'queue' to Kafka per se, but rather introduces cooperative consumption to accommodate these queuing use-cases, using regular Kafka topics."*
> — Apache Kafka 4.0 release notes

**Share groups don't turn the log into a queue. They add a queue-shaped way to read it.**

Acknowledged records are not deleted. The share group just moves its own starting position forward, and retention still decides how long records live. The same topic can serve consumer groups and share groups at the same time: your fraud detector replays history as a stream while a worker pool processes the same records as jobs. A share group is closer to a durable shared subscription than to SQS (and the clients are Java-first for now, check yours before planning a migration).

| | Consumer group | Share group |
|---|---|---|
| Partition assignment | Exclusive, one consumer per partition | None, records dispatched to any consumer |
| Max parallelism | Partition count | Consumer count |
| Acknowledgement | Offset commit | Per record: accept, release, reject |
| Retry | Replay by seeking offsets | Automatic redelivery, capped attempts |
| Ordering | Guaranteed per partition | None |
| Delivery | At-least-once, or exactly-once with transactions | At-least-once |

## The right mental model

Kafka is a distributed, replayable log. A consumer group reads it as a stream: ordered per key, replayable, stateful. A share group reads it as a work pool: unordered, acknowledged record by record. Two read models, one storage model, and the storage model is the part that never changed.

So design for the log (partition counts, keys, retention) and pick the read model per use case. The old reflex (single partitions for ordering, short retention to clean up, alerts on absolute lag) doesn't work any better with share groups around. Kafka grew a queue-shaped door, but the building is still a log.

[Book a demo](https://www.conduktor.io/contact/demo) to see how Conduktor gives you visibility into every consumer group, lag trend, and topic across your Kafka estate.

---

**Related**: [Event Sourcing with Kafka →](https://www.conduktor.io/blog/event-sourcing-kafka-patterns-pitfalls) · [Exactly-Once Semantics, When They Work →](https://www.conduktor.io/blog/exactly-once-semantics-when-it-works) · [Dead Letter Topics →](https://www.conduktor.io/blog/dead-letter-topics-handling-poison-pills)