# Kafka vs Postgres: When to Use Each

**Kafka** is a distributed append-only log built for retention, replay, and many independent consumers.

**Postgres** is an ACID transactional table store that can also act as a durable queue via `SELECT ... FOR UPDATE SKIP LOCKED`.

The Kafka vs Postgres choice usually comes down to scale, architecture, and ownership of the systems. Most workloads sit well below the scale Kafka is designed for, which raises a fair question: why not just use Postgres?

A quick TL;DR: it's rarely about technology; it's about your goal. As a queue or a single application's backend, Postgres is the right call. For **data exchange at scale**, sharing data across many teams, services, or organizations, Postgres is a definitive no-go, for architectural reasons rather than performance. That is the heart of the matter: [why Kafka is the classic choice for organizational data exchange](#why-kafka-is-the-classic-choice-for-organizational-data-exchange).

![kafka-vs-postgres diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-vs-postgres-0.webp)

## What each system actually is

- [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) is a distributed commit log. Records are appended to partitioned, replicated [topics](https://www.conduktor.io/glossary/kafka-topics-explained) and retained by a policy (time, size, or compaction); they are **not deleted when read**. Consumers pull at their own offset, and multiple consumer groups read the same topic independently. This is what makes replay and fan-out first-class primitives.

- Postgres is an ACID transactional database. Its primitives are tables, rows, MVCC snapshots, and the write-ahead log (WAL). It is durable like Kafka, but the durability serves transactions and point-in-time consistency, not a re-readable event history. A queue on Postgres is just a table with a `status` column and the right locking.

Both are durable. The difference is the shape of the data: an immutable log to re-read versus a mutable set of rows to claim and remove.

Both can act as pub/sub or queues.

## Kafka vs Postgres at a glance

| Dimension | Apache Kafka | PostgreSQL (as a queue) |
|---|---|---|
| Storage model | Distributed append-only commit log | ACID row store (queue table) |
| Retention / replay | Retained by time/size; replay any offset | Rows deleted on ack; no replay after delete |
| Consumer fan-out | N independent consumer groups, each a full copy | One worker pool per table (competing consumers) |
| Throughput ceiling | Millions of msg/s across a cluster | Thousands of msg/s single node; scales with cores |
| Ordering | Per-partition (per key) | Per single reader; lost across competing workers |
| Delivery | At-least-once; exactly-once with transactions | At-least-once (claim-and-delete) |
| Failover | Automatic, replicated brokers | Single writer; needs Patroni or manual promotion |
| Ops cost | Cluster + Connect + Schema Registry | One database, very common |
| Best fit | High throughput, replay, many readers | Small-to-medium queues, one worker pool |

## Postgres as a queue: SKIP LOCKED, LISTEN/NOTIFY, pgmq

The competing-consumers pattern on Postgres relies on `FOR UPDATE SKIP LOCKED`. Each worker locks a batch of `ready` rows; `SKIP LOCKED` makes concurrent workers step over rows another transaction already holds.

You will want a partial index to avoid a full table scan on `WHERE status = 'ready' ORDER BY id`:

```sql
CREATE INDEX jobs_ready_idx ON jobs (id) WHERE status = 'ready';
```

You need to claim and mark rows atomically in one statement, with a CTE that selects `FOR UPDATE SKIP LOCKED` feeding an `UPDATE ... RETURNING`:

```sql
-- each worker runs, in its own transaction:
BEGIN;
WITH c AS (
  SELECT id FROM jobs
  WHERE status = 'ready'
  ORDER BY id
  FOR UPDATE SKIP LOCKED
  LIMIT 3
)
UPDATE jobs SET status = 'workerA'
FROM c WHERE jobs.id = c.id
RETURNING jobs.id;
COMMIT;
```

For instance, that would give:

```text
Worker A RETURNING: 1,2,3
Worker B RETURNING (returned immediately while A held locks): 4,5,6

SELECT status, count(*), array_agg(id ORDER BY id) FROM jobs GROUP BY status:
 ready   | 4 | {7,8,9,10}
 workerA | 3 | {1,2,3}
 workerB | 3 | {4,5,6}
```

Without `SKIP LOCKED`, a plain `SELECT ... FOR UPDATE` from the second worker would **block** on the rows the first transaction holds.

You can:

- *claim-and-delete* to remove the rows on ack
- *claim-and-mark* flips a `status` column

Marking is easier to audit but leaves rows in place, which compounds the MVCC bloat covered below.

Then there is **crash recovery**. If a worker dies mid-job, its rows stay stuck in the in-flight state and are never reprocessed.

Therefore, you may need to add a `claimed_at timestamptz` and a periodic reaper that resets abandoned rows:

```sql
-- reaper: reclaim rows a crashed worker never finished
UPDATE jobs SET status = 'ready', claimed_at = NULL
WHERE status = 'processing' AND claimed_at < now() - interval '5 minutes';
```

To avoid constant polling, workers can use **LISTEN/NOTIFY** to add push-style wake-ups. A worker runs `LISTEN jobs_channel` and blocks, an insert fires `NOTIFY jobs_channel`:

```sql
LISTEN jobs_channel;                 -- worker session
NOTIFY jobs_channel, 'new job';      -- producer, after INSERT
```

This has a big drawback: if no listener is connected when `NOTIFY` fires, the notification is lost (there is no replay) and very high notification rates can contend on an internal lock.

**pgmq** is a PostgreSQL extension (originally from Tembo) that packages all of this behind an SQS-like SQL API. It uses `FOR UPDATE SKIP LOCKED` internally and is the recommended path over hand-rolling queue machinery:

```sql
SELECT pgmq.send('jobs', '{"id": 1}');   -- enqueue
SELECT * FROM pgmq.read('jobs', 30, 1);  -- claim with a 30s visibility timeout
SELECT pgmq.archive('jobs', 1);          -- ack: move to archive table
```

The cost of any of these is **MVCC bloat**. Even `pgmq.read()` performs an `UPDATE` (to move the visibility timeout), which generates dead tuples. On a hot queue table, autovacuum often *cannot* keep pace, so you need aggressive per-table autovacuum settings.

## The real debate: most Kafka carries small data

It's common in the community to say "you don't need Kafka, just use Postgres". If you have a workload of 500 KB/s, you don't need a Kafka cluster: a queue table (or pgmq) on a database already in production is simpler to run.

Even Aiven and Redpanda confirmed that roughly **50% of deployments ingest below 10 MB/s**. This is equivalent to 1000 msg/s (for an average of 10KB per message). A well-tuned Postgres can easily cover that.

## Where Kafka still earns its place

Postgres-as-a-queue stops being enough when you need real scale and fan-out without heavy tuning. Kafka's advantages here are structural, not incremental:

- **Replay and log semantics.** A consumer can reset its offset and re-read every retained record. A simple Postgres queue deletes or archives rows on ack, so replay is not built in: it requires explicit retention plus per-consumer position tracking, versus Kafka's retained offsets. Kafka offers both queue-like consumption (via [share groups](https://www.conduktor.io/glossary/kafka-share-groups)) and a durable, replayable log; a queue table gives only the former.
- **Fan-out to many independent consumers.** Multiple consumer groups each read the full stream at their own pace, so search indexing, analytics, caches, and other services consume the same events without coordinating. A queue table feeds one worker pool; adding independent readers means extra modeling, such as per-consumer checkpoints, logical-replication publications, or separate read models.
- **Specialized, multi-language clients.** Kafka has a mature multi-language client ecosystem: the JVM client maintained by Apache Kafka plus widely used Go, Python, Rust, C/C++, and .NET clients. A consumer subscribes and receives records, with no query to write. Postgres access means SQL plus a driver, and each consumer hand-rolls the claim, ack, and reaper logic (or adopts pgmq).
- **No MVCC bloat.** Kafka's log is append-only, so old data ages out by retention rather than by vacuuming dead tuples. A busy Postgres queue churns rows constantly (every claim, and even `pgmq.read()`, is an UPDATE), making autovacuum a permanent operational concern.
- **Native horizontal scaling and failover.** Scaling Kafka is vanilla: add partitions and brokers and throughput grows, with automatic failover across replicated brokers. Postgres has built-in streaming replication and read replicas, but write scaling is vertical by default: sharding and automatic failover orchestration (for example Patroni) are not core and rely on external tooling.

## Why Kafka is the classic choice for organizational data exchange

A database shared *directly* across teams tends to expose far more of its internal surface than intended: arbitrary reads and writes, joins over internal tables, and a physical schema consumers start to depend on.

Postgres can narrow that surface with roles, column privileges, views, row-level security, and read replicas, but each is deliberate work that gets harder to maintain as more teams connect.

The table schema becomes a public API: no team can rename a column or refactor a table without risking someone else's query, and every direct reader adds query load and connection pressure to the OLTP path unless reads are offloaded to a replica.

**This is the well-known shared-database (integration-database) anti-pattern.**

Kafka exposes a deliberately narrow interface: producers append immutable events, consumers subscribe and read, nothing more. That restriction is the point:

- **Read-only for consumers.** A subscriber reads events; it cannot write to or run a query against the producer's database.
- **Schema as an explicit contract.** The event schema is the interface, versioned with compatibility rules ([data contracts](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines), [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management)).
- **Decoupling in time and ownership.** Producers and consumers do not know each other. A new team subscribes without anything to change on the producer side.
- **Governance at the right granularity.** [ACLs](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) and quotas grant "subscribe to the orders stream," not "read our database," with per-client audit and rate limits.

A shared log is a much cleaner backbone for cross-team and [cross-organization data sharing](https://www.conduktor.io/glossary/cross-organization-data-sharing-patterns), and a foundation of [data mesh](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation).

## They are complementary: Postgres + Debezium CDC into Kafka

This is a very common pattern: Postgres stays the ACID source of truth for microservices, and [Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) reads its WAL to turn every insert, update, and delete into [change events](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) on Kafka topics.

Downstream, many independent consumers read from Kafka directly (search indexing, analytics, caches, other services) without touching the OLTP database:

![kafka-vs-postgres diagram 2](https://www.conduktor.io/assets/images/glossary/kafka-vs-postgres-1.webp)

**Can Postgres replace Kafka for a message queue?**

For a single worker pool below a single node's throughput, usually yes. `FOR UPDATE SKIP LOCKED` (or pgmq) gives real competing-consumers semantics on a database you already run. It cannot give you replay or independent fan-out to many consumer groups.

**Is SKIP LOCKED safe for concurrent workers?**

Yes. Tested on PostgreSQL 16.14, two concurrent workers claimed disjoint row sets ({1,2,3} and {4,5,6}) and the second never blocked. The safe pattern claims and marks rows atomically in one CTE feeding an UPDATE ... RETURNING.

**What is the main downside of a Postgres queue?**

MVCC bloat. Every claim and even pgmq.read() is an UPDATE that creates dead tuples; 200 reads generated 200 dead tuples in testing. Autovacuum must keep up, and short benchmarks hide the pile-up.

**Does Kafka now support queue semantics?**

Yes, via KIP-932 share groups: GA in Kafka 4.2, after early access in 4.0 and preview in 4.1. It adds per-message acknowledgement and message-level parallelism.

**How do Postgres and Kafka work together?**

Postgres stays the transactional source of truth and Debezium captures its WAL into Kafka topics. Tools like [Conduktor](https://www.conduktor.io/kafka-data-governance) then help teams observe and govern those topics as they fan out to many downstream consumers.

## Related Pages

- [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka): the distributed log at the center of the comparison.
- [Kafka vs RabbitMQ](https://www.conduktor.io/glossary/kafka-vs-rabbitmq): streams versus message queues, the sibling comparison.
- [Change Data Capture (CDC) Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals): how database changes become event streams.
- [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium): capturing the Postgres WAL into Kafka topics.
- [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture): the pattern fan-out and replay enable.
- [Streaming ETL vs Traditional ETL](https://www.conduktor.io/glossary/streaming-etl-vs-traditional-etl): why a shared log changes the pipeline.

## Sources

- [KIP-932: Queues for Kafka, Apache Kafka wiki](https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka)
- [Queues for Kafka (KIP-932), Preview Release Notes](https://cwiki.apache.org/confluence/display/KAFKA/Queues+for+Kafka+(KIP-932)+-+Preview+Release+Notes)
- [Apache Kafka 4.0.0 Release Announcement](https://kafka.apache.org/blog/2025/03/18/apache-kafka-4.0.0-release-announcement/)
- [Kafka Queue Semantics Now GA with Share Consumer API, Confluent](https://www.confluent.io/blog/kafka-queue-semantics-share-consumer-ga/)
- [PostgreSQL Documentation: SELECT, The Locking Clause (FOR UPDATE / SKIP LOCKED)](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE)
- [Message Queuing Using Native PostgreSQL, Crunchy Data](https://www.crunchydata.com/blog/message-queuing-using-native-postgresql)
- ["You Don't Need Kafka, Just Use Postgres" Considered Harmful, Gunnar Morling](https://www.morling.dev/blog/you-dont-need-kafka-just-use-postgres-considered-harmful/)
- [Kafka is fast, I'll use Postgres (benchmarks), TopicPartition](https://topicpartition.io/blog/postgres-pubsub-queue-benchmarks)
- [Postgres, Kafka and event queues, Kaarel Moppel](https://kmoppel.github.io/2025-11-13-postgres-kafka-and-event-queues/)
- [pgmq: A lightweight message queue for PostgreSQL (GitHub)](https://github.com/pgmq/pgmq)
- [Debezium Documentation: PostgreSQL connector](https://debezium.io/documentation/reference/stable/connectors/postgresql.html)