Kafka vs Postgres: When to Use Each

Stéphane Derosiaux July 18, 2026 7 min read

Kafka is a distributed append-only log built for retention, replay, and many independent consumers.

Postgres is an ACID transactional table store that can also act as a durable queue via SELECT ... FOR UPDATE SKIP LOCKED.

The Kafka vs Postgres choice usually comes down to scale, architecture, and ownership of the systems. Most workloads sit well below the scale Kafka is designed for, which raises a fair question: why not just use Postgres?

A quick TL;DR: it's rarely about technology; it's about your goal. As a queue or a single application's backend, Postgres is the right call. For data exchange at scale, sharing data across many teams, services, or organizations, Postgres is a definitive no-go, for architectural reasons rather than performance. That is the heart of the matter: why Kafka is the classic choice for organizational data exchange.

kafka-vs-postgres diagram 1

What each system actually is

  • Apache Kafka is a distributed commit log. Records are appended to partitioned, replicated topics and retained by a policy (time, size, or compaction); they are not deleted when read. Consumers pull at their own offset, and multiple consumer groups read the same topic independently. This is what makes replay and fan-out first-class primitives.
  • Postgres is an ACID transactional database. Its primitives are tables, rows, MVCC snapshots, and the write-ahead log (WAL). It is durable like Kafka, but the durability serves transactions and point-in-time consistency, not a re-readable event history. A queue on Postgres is just a table with a status column and the right locking.

Both are durable. The difference is the shape of the data: an immutable log to re-read versus a mutable set of rows to claim and remove.

Both can act as pub/sub or queues.

Kafka vs Postgres at a glance

DimensionApache KafkaPostgreSQL (as a queue)
Storage modelDistributed append-only commit logACID row store (queue table)
Retention / replayRetained by time/size; replay any offsetRows deleted on ack; no replay after delete
Consumer fan-outN independent consumer groups, each a full copyOne worker pool per table (competing consumers)
Throughput ceilingMillions of msg/s across a clusterThousands of msg/s single node; scales with cores
OrderingPer-partition (per key)Per single reader; lost across competing workers
DeliveryAt-least-once; exactly-once with transactionsAt-least-once (claim-and-delete)
FailoverAutomatic, replicated brokersSingle writer; needs Patroni or manual promotion
Ops costCluster + Connect + Schema RegistryOne database, very common
Best fitHigh throughput, replay, many readersSmall-to-medium queues, one worker pool

Postgres as a queue: SKIP LOCKED, LISTEN/NOTIFY, pgmq

The competing-consumers pattern on Postgres relies on FOR UPDATE SKIP LOCKED. Each worker locks a batch of ready rows; SKIP LOCKED makes concurrent workers step over rows another transaction already holds.

You will want a partial index to avoid a full table scan on WHERE status = 'ready' ORDER BY id:

CREATE INDEX jobs_ready_idx ON jobs (id) WHERE status = 'ready';

You need to claim and mark rows atomically in one statement, with a CTE that selects FOR UPDATE SKIP LOCKED feeding an UPDATE ... RETURNING:

-- each worker runs, in its own transaction:
BEGIN;
WITH c AS (
  SELECT id FROM jobs
  WHERE status = 'ready'
  ORDER BY id
  FOR UPDATE SKIP LOCKED
  LIMIT 3
)
UPDATE jobs SET status = 'workerA'
FROM c WHERE jobs.id = c.id
RETURNING jobs.id;
COMMIT;

For instance, that would give:

Worker A RETURNING: 1,2,3
Worker B RETURNING (returned immediately while A held locks): 4,5,6

SELECT status, count(*), array_agg(id ORDER BY id) FROM jobs GROUP BY status:
 ready   | 4 | {7,8,9,10}
 workerA | 3 | {1,2,3}
 workerB | 3 | {4,5,6}

Without SKIP LOCKED, a plain SELECT ... FOR UPDATE from the second worker would block on the rows the first transaction holds.

You can:

  • claim-and-delete to remove the rows on ack
  • claim-and-mark flips a status column

Marking is easier to audit but leaves rows in place, which compounds the MVCC bloat covered below.

Then there is crash recovery. If a worker dies mid-job, its rows stay stuck in the in-flight state and are never reprocessed.

Therefore, you may need to add a claimed_at timestamptz and a periodic reaper that resets abandoned rows:

-- reaper: reclaim rows a crashed worker never finished
UPDATE jobs SET status = 'ready', claimed_at = NULL
WHERE status = 'processing' AND claimed_at < NOW() - interval '5 minutes';

To avoid constant polling, workers can use LISTEN/NOTIFY to add push-style wake-ups. A worker runs LISTEN jobs_channel and blocks, an insert fires NOTIFY jobs_channel:

LISTEN jobs_channel;                 -- worker session
NOTIFY jobs_channel, 'new job';      -- producer, after INSERT

This has a big drawback: if no listener is connected when NOTIFY fires, the notification is lost (there is no replay) and very high notification rates can contend on an internal lock.

pgmq is a PostgreSQL extension (originally from Tembo) that packages all of this behind an SQS-like SQL API. It uses FOR UPDATE SKIP LOCKED internally and is the recommended path over hand-rolling queue machinery:

SELECT pgmq.send('jobs', '{"id": 1}');   -- enqueue
SELECT * FROM pgmq.read('jobs', 30, 1);  -- claim with a 30s visibility timeout
SELECT pgmq.archive('jobs', 1);          -- ack: move to archive table

The cost of any of these is MVCC bloat. Even pgmq.read() performs an UPDATE (to move the visibility timeout), which generates dead tuples. On a hot queue table, autovacuum often cannot keep pace, so you need aggressive per-table autovacuum settings.

The real debate: most Kafka carries small data

It's common in the community to say "you don't need Kafka, just use Postgres". If you have a workload of 500 KB/s, you don't need a Kafka cluster: a queue table (or pgmq) on a database already in production is simpler to run.

Even Aiven and Redpanda confirmed that roughly 50% of deployments ingest below 10 MB/s. This is equivalent to 1000 msg/s (for an average of 10KB per message). A well-tuned Postgres can easily cover that.

Where Kafka still earns its place

Postgres-as-a-queue stops being enough when you need real scale and fan-out without heavy tuning. Kafka's advantages here are structural, not incremental:

  • Replay and log semantics. A consumer can reset its offset and re-read every retained record. A simple Postgres queue deletes or archives rows on ack, so replay is not built in: it requires explicit retention plus per-consumer position tracking, versus Kafka's retained offsets. Kafka offers both queue-like consumption (via share groups) and a durable, replayable log; a queue table gives only the former.
  • Fan-out to many independent consumers. Multiple consumer groups each read the full stream at their own pace, so search indexing, analytics, caches, and other services consume the same events without coordinating. A queue table feeds one worker pool; adding independent readers means extra modeling, such as per-consumer checkpoints, logical-replication publications, or separate read models.
  • Specialized, multi-language clients. Kafka has a mature multi-language client ecosystem: the JVM client maintained by Apache Kafka plus widely used Go, Python, Rust, C/C++, and .NET clients. A consumer subscribes and receives records, with no query to write. Postgres access means SQL plus a driver, and each consumer hand-rolls the claim, ack, and reaper logic (or adopts pgmq).
  • No MVCC bloat. Kafka's log is append-only, so old data ages out by retention rather than by vacuuming dead tuples. A busy Postgres queue churns rows constantly (every claim, and even pgmq.read(), is an UPDATE), making autovacuum a permanent operational concern.
  • Native horizontal scaling and failover. Scaling Kafka is vanilla: add partitions and brokers and throughput grows, with automatic failover across replicated brokers. Postgres has built-in streaming replication and read replicas, but write scaling is vertical by default: sharding and automatic failover orchestration (for example Patroni) are not core and rely on external tooling.

Why Kafka is the classic choice for organizational data exchange

A database shared directly across teams tends to expose far more of its internal surface than intended: arbitrary reads and writes, joins over internal tables, and a physical schema consumers start to depend on.

Postgres can narrow that surface with roles, column privileges, views, row-level security, and read replicas, but each is deliberate work that gets harder to maintain as more teams connect.

The table schema becomes a public API: no team can rename a column or refactor a table without risking someone else's query, and every direct reader adds query load and connection pressure to the OLTP path unless reads are offloaded to a replica.

This is the well-known shared-database (integration-database) anti-pattern.

Kafka exposes a deliberately narrow interface: producers append immutable events, consumers subscribe and read, nothing more. That restriction is the point:

  • Read-only for consumers. A subscriber reads events; it cannot write to or run a query against the producer's database.
  • Schema as an explicit contract. The event schema is the interface, versioned with compatibility rules (data contracts, Schema Registry).
  • Decoupling in time and ownership. Producers and consumers do not know each other. A new team subscribes without anything to change on the producer side.
  • Governance at the right granularity. ACLs and quotas grant "subscribe to the orders stream," not "read our database," with per-client audit and rate limits.

A shared log is a much cleaner backbone for cross-team and cross-organization data sharing, and a foundation of data mesh.

They are complementary: Postgres + Debezium CDC into Kafka

This is a very common pattern: Postgres stays the ACID source of truth for microservices, and Debezium reads its WAL to turn every insert, update, and delete into change events on Kafka topics.

Downstream, many independent consumers read from Kafka directly (search indexing, analytics, caches, other services) without touching the OLTP database:

kafka-vs-postgres diagram 2

Can Postgres replace Kafka for a message queue?

For a single worker pool below a single node's throughput, usually yes. FOR UPDATE SKIP LOCKED (or pgmq) gives real competing-consumers semantics on a database you already run. It cannot give you replay or independent fan-out to many consumer groups.

Is SKIP LOCKED safe for concurrent workers?

Yes. Tested on PostgreSQL 16.14, two concurrent workers claimed disjoint row sets ({1,2,3} and {4,5,6}) and the second never blocked. The safe pattern claims and marks rows atomically in one CTE feeding an UPDATE ... RETURNING.

What is the main downside of a Postgres queue?

MVCC bloat. Every claim and even pgmq.read() is an UPDATE that creates dead tuples; 200 reads generated 200 dead tuples in testing. Autovacuum must keep up, and short benchmarks hide the pile-up.

Does Kafka now support queue semantics?

Yes, via KIP-932 share groups: GA in Kafka 4.2, after early access in 4.0 and preview in 4.1. It adds per-message acknowledgement and message-level parallelism.

How do Postgres and Kafka work together?

Postgres stays the transactional source of truth and Debezium captures its WAL into Kafka topics. Tools like Conduktor then help teams observe and govern those topics as they fan out to many downstream consumers.

Sources