Kafka vs RabbitMQ: Streams vs Queues

Apache Kafka is a distributed log for stream storage and replay: records are retained durably and consumers pull at their own pace. RabbitMQ is a message broker implementing AMQP: messages are pushed to consumers and deleted once acknowledged. The core difference is persistence model — Kafka is a log you can re-read; RabbitMQ is a queue that empties as messages are consumed.

TL;DR

What is Apache Kafka?

Apache Kafka is a distributed event streaming platform. Records are written to named topics, stored in partitioned, replicated logs on broker disk, and retained independently of consumption. Consumer groups track their position (offset) in each partition; multiple groups can independently read the same topic. Records are never deleted on read — only by retention policy (time or size).

What is RabbitMQ?

RabbitMQ is a message broker developed by Rabbit Technologies (now VMware/Broadcom), written in Erlang. It implements AMQP 0-9-1 and supports additional protocols (STOMP, MQTT, HTTP). Producers publish messages to exchanges; exchanges route to queues based on routing rules. Consumers subscribe to queues and receive messages via push delivery. Once a consumer acknowledges a message, the broker removes it.

RabbitMQ 3.9 introduced streams (via the rabbitmq_stream plugin), which add an append-only log structure with offset-based consumption closer to Kafka's model. However, this is a separate protocol and not the default queue behavior.

Architecture compared

Storage and delivery

Kafka stores every message on disk regardless of consumer state. A message written at 9:00 AM is still readable at 9:00 PM (within the retention window). Consumers self-manage their read position (offset). This makes Kafka suitable for:

• Event sourcing: replay history to rebuild state
• Multiple independent consumers: analytics, alerting, and replication can all read the same topic
• Audit logs: immutable record of what happened and when

RabbitMQ's queues are transient by nature: messages exist to be delivered and then removed once acknowledged. (RabbitMQ Streams, available since 3.9, behave differently — they are append-only and non-destructive.) Queue depth for classic/quorum queues is bounded by memory/disk limits, not a durability window. This makes RabbitMQ suitable for:

• Task queues: distribute units of work to worker processes
• RPC patterns: request-reply with correlation IDs
• Short-lived messages: where post-delivery persistence adds no value

Consumer model

Kafka consumer groups: Partitions are assigned to consumers in a group. Each partition is consumed by exactly one consumer within the group at a time; a single consumer may own multiple partitions, and if there are more consumers than partitions, some consumers sit idle. Adding consumers scales throughput up to the partition count. Multiple independent groups read the same topic without interfering. See Kafka Consumer Groups Explained.

RabbitMQ competing consumers: Multiple consumers on the same queue share messages round-robin (one consumer per message). This is a classic work-queue pattern but does not allow multiple independent consumers to each receive a copy of the same message without separate queues (fanout exchanges route copies to multiple queues, approximating Kafka's multi-group model).

Routing

RabbitMQ provides rich routing via exchange types:

• Direct: route by exact routing key
• Fanout: broadcast to all bound queues
• Topic: wildcard routing key patterns
• Headers: route by message header attributes

Kafka routes by producer-specified topic name. Filtering within a topic requires consumer-side processing or KafkaStreams/Flink. The simplicity is intentional — Kafka offloads routing logic to producers and consumers.

Ordering

Kafka guarantees ordering within a partition. Messages with the same key always go to the same partition (key-based routing), ensuring ordered delivery per entity (e.g., all events for user ID 123 are ordered). Cross-partition ordering is not guaranteed.

RabbitMQ guarantees ordering within a single queue with a single consumer. With competing consumers, message ordering is not preserved (a slow consumer can hold an unacked message while faster consumers process later messages).

Operational trade-offs

Kafka advantages:

• High throughput at scale — partitioned design allows horizontal scaling without coordination
• Durable replay — essential for event sourcing, CDC pipelines, and audit requirements
• Multiple independent consumer groups at no extra cost
• Native support for stream processing via Kafka Streams or Flink on top

Kafka disadvantages:

• Higher operational complexity — partition management, consumer group rebalancing, offset tracking
• No built-in complex routing logic — routing must be in producers or external processors
• Messages cannot be selectively deleted (only log compaction per key)
• Not suitable as a task queue where you need exactly-one delivery without re-reads

RabbitMQ advantages:

• Rich routing model (exchanges, binding keys) with minimal client code
• Lower latency for small message volumes — push delivery is faster for interactive workloads
• Message TTL, dead-letter queues, priority queues built-in
• Easier to reason about for simple point-to-point or worker-queue patterns
• Per-message acknowledgment and requeue semantics

RabbitMQ disadvantages:

• Messages are lost after acknowledgment — no replay, no audit trail by default
• Queue depth is memory-bounded; large queues degrade performance
• Competing consumers don't scale as cleanly for high-throughput partitioned workloads
• No native stream processing integration

When to choose Kafka

• You need replay: a new service needs to process historical events without data loss
• You have multiple independent consumers of the same event stream
• You are building CDC pipelines, event sourcing, or audit logging — retention is mandatory
• You need high throughput (millions of messages/sec) with predictable horizontal scaling
• Your data engineering stack uses tools that integrate natively with Kafka (Flink, Spark, dbt, etc.)
• You need exactly-once semantics in a transactional pipeline

When to choose RabbitMQ

• You have a task queue pattern: distribute units of work to workers, acknowledge on completion
• You need flexible routing — topic exchanges with wildcard routing keys, header-based dispatch
• Your messages are short-lived and replay is not a requirement
• You need per-message TTL or dead-letter exchange routing out of the box
• Your stack is polyglot and needs AMQP, STOMP, or MQTT protocol support
• You are implementing RPC or request-reply patterns with correlation IDs

Can Kafka and RabbitMQ coexist?

Yes — and this is common. A typical pattern: Kafka carries high-volume event streams (user activity, transactions), while RabbitMQ handles internal task distribution (send email, resize image, trigger workflow). They solve different problems and the boundary is usually clear: if you need replay or multiple consumers, use Kafka; if you need flexible task routing with acknowledgment semantics, use RabbitMQ.

Conduktor Console: Free Kafka platform for teams. Install in 5 minutes. Explore Conduktor Console →