You Probably Don't Need Kafka's Exactly-Once Semantics

I've seen teams enable exactly-once semantics (EOS) because "duplicates are bad" without measuring the cost. Then they wonder why throughput dropped 40% and commit latency tripled.

The uncomfortable truth: most applications don't need EOS. At-least-once with idempotent consumers is simpler, faster, and sufficient for nearly every real-world use case.

We enabled transactions for 'safety.' Throughput dropped from 100k to 60k msg/s. Then we realized our consumers write to PostgreSQL anyway—EOS didn't help. We switched to idempotent consumers and got our performance back.

Platform Engineer at a payments company

The Hidden Cost

Kafka's transactional exactly-once has overhead rarely discussed in marketing:

Mode	Throughput	Latency p99
At-least-once (acks=all)	100k msg/s	15ms
Idempotent producer	98k msg/s	16ms
Transactional	60k msg/s	45ms

That's a 40% throughput reduction. Every transaction requires coordination with __transaction_state. Commit latency adds 10-50ms per batch.

The read_committed consumer tax: consumers using isolation.level=read_committed can't advance past the oldest open transaction. One stuck producer blocks all consumers on affected partitions for transaction.timeout.ms (default: 60 seconds). Consumer group monitoring surfaces these stalls before they cascade.

Cascading failure risk: If many producers across many partitions encounter errors simultaneously (network partition, downstream outage), your entire consumer fleet can stall. This cascade has caused multi-hour outages. Always set transaction.timeout.ms ≤ 15 seconds in production and monitor transaction coordinator lag.

The Real Exactly-Once Boundary

Kafka's EOS guarantees atomicity for:

• Writes to multiple partitions
• Consumer offset commits
• Kafka Streams state stores

It does NOT guarantee exactly-once for:

• Database writes
• REST API calls
• Any external system

producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", key, order));
httpClient.post("https://payments.example.com/charge", order);  // NOT transactional
producer.commitTransaction();
// If HTTP succeeded but commit fails: customer charged, no Kafka record

If your consumer writes to PostgreSQL, Redis, or any external system, EOS doesn't help. You need idempotent consumers anyway.

The Idempotent Consumer Alternative

An idempotent consumer produces the same result whether a message is processed once or multiple times. This works for 95% of use cases where you write to external systems.

public void processOrder(ConsumerRecord<String, OrderEvent> record) {
    jdbcTemplate.update("""
        INSERT INTO orders (order_id, customer_id, total, status)
        VALUES (?, ?, ?, ?)
        ON CONFLICT (order_id) DO NOTHING
        """,
        record.value().getOrderId(),
        record.value().getCustomerId(),
        record.value().getTotal(), "PENDING");
}

Process the same message 10 times? One row in the database. No transactions, no coordinator overhead, no blocked consumers.

Design for idempotency:

-- State replacement (idempotent)
UPDATE inventory SET quantity = 50 WHERE product_id = 'SKU-123';

-- Delta operation (NOT idempotent)
UPDATE inventory SET quantity = quantity - 1 WHERE product_id = 'SKU-123';

Carry full state in events, not deltas. Duplicates become harmless.

When EOS Actually Matters

Enable exactly-once when ALL of these are true:

1. Pure Kafka-to-Kafka processing — no external systems
2. Atomic multi-partition writes required — one message fans out to several topics
3. Duplicates break business logic — can't be handled by idempotent consumers
4. Latency budget permits — you can absorb 10-50ms commit overhead

The legitimate use case: Kafka Streams with stateful processing:

props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

This keeps state stores consistent with output topics. Kafka Streams handles the complexity internally.

Decision Framework

Situation	Recommendation
Writing to database	Idempotent consumer
Calling external APIs	Idempotent consumer
Kafka Streams with state	exactly_once_v2
Multi-topic atomic writes	Transactions
Logging, metrics, analytics	At-least-once
"Duplicates are bad"	Idempotent consumer

The last row is intentional. "Duplicates are bad" isn't a reason to use EOS. It's a reason to make your consumers idempotent.

Stop reaching for exactly-once as the default. Start with at-least-once and idempotent consumers. Add EOS only when you've proven you need it and can afford the overhead.

Book a demo to see how Conduktor Console shows transaction states and helps identify when EOS overhead isn't paying off.