Kafka Isn't a Queue: Stop Designing It Like One

How does a message leave RabbitMQ or SQS? A worker grabs it, acks it, the broker deletes it. Gone. That lifecycle is so universal that most of us carry it into Kafka without noticing, and then design topics around a deletion that never happens.

It shows up the same way in cluster after cluster: single-partition topics "for ordering", aggressive retention "to clean up after ourselves", consumer lag treated as an incident. None of these are Kafka problems. They are queue habits applied to a log.

Update, June 2026. Kafka now ships queue semantics: share groups (KIP-932) went early access in 4.0, preview in 4.1, and are production-ready since Apache Kafka 4.2 (February 2026). Share groups change how you read the log, not what Kafka is, so the mental model below still decides your design. Covered in detail near the end.

The queue mental model

A traditional queue works like a mailbox:

1. Producer drops a message
2. A consumer picks it up
3. The message gets deleted
4. The next consumer gets the next message

Multiple workers race to grab messages from a shared pool. Consumption is destructive: once processed, the message no longer exists. The broker tracks who has what, retries what failed, and the backlog physically shrinks as you work.

🚫 "Kafka is basically a faster RabbitMQ."

Kafka doesn't work this way.

Kafka is a log

Kafka is an append-only, immutable log:

Records are never deleted by consumers. They stay until retention expires (a storage policy, nothing to do with consumption). Each consumer tracks its own offset. Two consumers can read the same partition at the same time, at completely different positions, and neither affects the other.

Consuming changes nothing: the log stays, readers move.

What the log changes

Consuming deletes nothing

A bug in your processor? Fix it, reset offsets, rerun:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-processor --topic events \
  --reset-offsets --to-earliest --execute

This command makes no sense in a queue world (replay what? the messages are gone). In Kafka it's a standard recovery move, and it's the foundation of event sourcing patterns: the log is the source of truth, downstream state is derived and rebuildable.

Ordering is per-partition

Kafka guarantees ordering only within a partition. You can have total ordering or parallelism, not both:

Approach	Ordering	Parallelism
Single partition	Total order	1 consumer max
Key-based partitioning	Per-key order	Up to N consumers

Key-based partitioning is the sweet spot: all events for a given key (user ID, order ID) land on the same partition, in order, while other keys flow in parallel.

And design against keys, not partition numbers. The key names which entity must stay ordered, a business rule. The partition number is an infrastructure detail, and treating it as your ordering abstraction is a leak. For event-driven and event-sourced systems, I made the longer argument here.

Consumer groups divide work

In RabbitMQ, consumers compete for messages. In a Kafka consumer group, they split partitions:

Partition 0 → Consumer 1 (all records from P0)
Partition 1 → Consumer 2 (all records from P1)
Partition 2 → Consumer 3 (all records from P2)

Each partition belongs to exactly one consumer in the group. Add a fourth consumer to this topic and it sits idle.

➡ Partition count caps your parallelism. This is the constraint that pushed teams to over-partition for years, and the one share groups finally remove (more on this later).

Several groups, one topic

The part queues can't match:

Each group reads every record, independently, at its own pace. No fan-out exchange to configure, no data duplication, no upstream change when a fourth team shows up. You can monitor all your consumer groups in one place to keep track of who reads what.

The anti-patterns

Queue thinking produces the same four mistakes:

• Single partition "for ordering". You get total order, a hard ceiling of one consumer, and a painful recovery story. Key-based partitioning gives you the ordering you need (per key) without the ceiling.
• Short retention "to clean up". Retention is a storage policy, not a consumption policy. Nothing gets "cleaner" because consumers are done with the data. A week is common; a year isn't unusual for audit topics.
• Treating lag as a bug. Lag is information. 5,000 records behind could be a slow consumer (problem), an hourly batch job (expected), or a new consumer catching up on history (temporary). Alert on lag growth rate, not absolute lag.
• One topic per consumer. That's queue-world isolation. It duplicates data and couples producers to consumers. One topic, several consumer groups.

So if the log model is this good, why did the Kafka project just ship queues?

Share groups: queue semantics on the log (Kafka 4.2)

Because some workloads are queues. A pool of workers chewing through independent jobs (payment captures, image renders, emails) doesn't care about ordering. It cares about spreading records across as many workers as possible and retrying the ones that fail. For years the honest answer was "run SQS or RabbitMQ next to Kafka" (go figure), or over-partition so more consumers could attach.

KIP-932, "Queues for Kafka", is the project's answer. Share groups are a second way to consume a topic, production-ready since Kafka 4.2:

• No exclusive partition assignment: the broker hands records to whichever consumer asks

➡ consumers in a share group can outnumber partitions, so you stop over-partitioning just to add workers.

• Per-record acknowledgement: ACCEPT it, RELEASE it for redelivery (transient failure), or REJECT it (unprocessable)
• Fetched records are locked while you process them, 30 seconds by default

➡ no ack before the lock expires? The record goes back to the pool, delivery count +1. Kafka 4.2 added a way to extend the lock (RENEW) for slow records.

• Delivery attempts are capped, 5 by default

➡ past the cap the record is archived and skipped, not parked: there is no built-in dead letter queue yet (KIP-1191 is the proposal).

• Delivery is at-least-once, ordering across a worker pool is gone, and exactly-once semantics are not supported.

In code, it's a KafkaShareConsumer and acks per record instead of offset commits:

// consumer config: share.acknowledgement.mode=explicit
KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props);
consumer.subscribe(List.of("payment-jobs"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(200));
    for (ConsumerRecord<String, String> record : records) {
        try {
            process(record);
            consumer.acknowledge(record, AcknowledgeType.ACCEPT);
        } catch (RetriableException e) {
            consumer.acknowledge(record, AcknowledgeType.RELEASE); // back to the pool
        } catch (Exception e) {
            consumer.acknowledge(record, AcknowledgeType.REJECT);  // unprocessable
        }
    }
    consumer.commitSync(); // acknowledgements are sent here
}

Does that make Kafka a queue now? The release notes themselves are careful about this:

"It does not add the concept of a 'queue' to Kafka per se, but rather introduces cooperative consumption to accommodate these queuing use-cases, using regular Kafka topics." — Apache Kafka 4.0 release notes

Share groups don't turn the log into a queue. They add a queue-shaped way to read it.

Acknowledged records are not deleted. The share group just moves its own starting position forward, and retention still decides how long records live. The same topic can serve consumer groups and share groups at the same time: your fraud detector replays history as a stream while a worker pool processes the same records as jobs. A share group is closer to a durable shared subscription than to SQS (and the clients are Java-first for now, check yours before planning a migration).

	Consumer group	Share group
Partition assignment	Exclusive, one consumer per partition	None, records dispatched to any consumer
Max parallelism	Partition count	Consumer count
Acknowledgement	Offset commit	Per record: accept, release, reject
Retry	Replay by seeking offsets	Automatic redelivery, capped attempts
Ordering	Guaranteed per partition	None
Delivery	At-least-once, or exactly-once with transactions	At-least-once

The right mental model

Kafka is a distributed, replayable log. A consumer group reads it as a stream: ordered per key, replayable, stateful. A share group reads it as a work pool: unordered, acknowledged record by record. Two read models, one storage model, and the storage model is the part that never changed.

So design for the log (partition counts, keys, retention) and pick the read model per use case. The old reflex (single partitions for ordering, short retention to clean up, alerts on absolute lag) doesn't work any better with share groups around. Kafka grew a queue-shaped door, but the building is still a log.

Book a demo to see how Conduktor gives you visibility into every consumer group, lag trend, and topic across your Kafka estate.

---

Related: Event Sourcing with Kafka → · Exactly-Once Semantics, When They Work → · Dead Letter Topics →