When Kafka Exactly-Once Semantics Are Worth the Performance Cost

Teams often enable exactly-once semantics (EOS) because "duplicates are bad" without measuring the cost. Then they pay for it in end-to-end latency and stalled consumers.

The uncomfortable truth: most applications don't need EOS. At-least-once with idempotent consumers is simpler, faster, and sufficient for nearly every real-world use case.

The Hidden Cost

The throughput hit is smaller than most people fear. Confluent's original benchmark (2017, Kafka 0.11) measured:

Setup	Throughput vs baseline
Idempotent producer	Negligible impact
Transactional producer, 1 KB records, 100 ms transactions	-3% vs acks=all, -20% vs acks=1
Kafka Streams EOS, 100 ms commit interval	-15% (1 KB records) to -30% (100-byte records)
Kafka Streams EOS, 30 s commit interval, 1 KB+ records	No measurable overhead

Source: Exactly-once Semantics are Possible. Measure on your own workload; these numbers are old and setup-specific.

The cost is per transaction, not per record. Every transaction requires coordination with __transaction_state and a commit marker on each partition it touched, so small transactions pay the most. The bigger cost is latency: a read_committed consumer sees nothing until the transaction commits, so end-to-end latency grows with the transaction or commit interval.

The read_committed consumer tax: consumers using isolation.level=read_committed can't advance past the oldest open transaction. One stuck producer blocks all consumers on affected partitions for transaction.timeout.ms (default: 60 seconds). Consumer group monitoring surfaces these stalls before they cascade.

Cascading failure risk: If many producers across many partitions encounter errors simultaneously (network partition, downstream outage), your entire consumer fleet can stall. Keep transaction.timeout.ms close to how long your transactions actually take, so a stuck producer is aborted quickly (the broker caps it at transaction.max.timeout.ms, 15 minutes by default), and watch consumer lag on read_committed groups.

The Real Exactly-Once Boundary

Kafka's EOS guarantees atomicity for:

• Writes to multiple partitions
• Consumer offset commits
• Kafka Streams state stores

It does NOT guarantee exactly-once for:

• Database writes
• REST API calls
• Any external system

producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", key, order));
httpClient.post("https://payments.example.com/charge", order);  // NOT transactional
producer.commitTransaction();
// If HTTP succeeded but commit fails: customer charged, no Kafka record

If your consumer writes to PostgreSQL, Redis, or any external system, EOS doesn't help. You need idempotent consumers anyway.

The Idempotent Consumer Alternative

An idempotent consumer produces the same result whether a message is processed once or multiple times. This covers the common case where the consumer writes to an external system.

public void processOrder(ConsumerRecord<String, OrderEvent> record) {
    jdbcTemplate.update("""
        INSERT INTO orders (order_id, customer_id, total, status)
        VALUES (?, ?, ?, ?)
        ON CONFLICT (order_id) DO NOTHING
        """,
        record.value().getOrderId(),
        record.value().getCustomerId(),
        record.value().getTotal(), "PENDING");
}

Process the same message 10 times? One row in the database. No transactions, no coordinator overhead, no blocked consumers.

Design for idempotency:

-- State replacement (idempotent)
UPDATE inventory SET quantity = 50 WHERE product_id = 'SKU-123';

-- Delta operation (NOT idempotent)
UPDATE inventory SET quantity = quantity - 1 WHERE product_id = 'SKU-123';

Carry full state in events, not deltas. Duplicates become harmless. Add a version column and update only when the incoming version is newer, so a late replay of an old event can't overwrite fresher state.

When EOS Actually Matters

Enable exactly-once when ALL of these are true:

1. Pure Kafka-to-Kafka processing: no external systems
2. Atomic multi-partition writes required: one message fans out to several topics
3. Duplicates break business logic: can't be handled by idempotent consumers
4. Latency budget permits: downstream consumers can wait for each transaction to commit

The legitimate use case: Kafka Streams with stateful processing:

props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

This keeps state stores consistent with output topics. Kafka Streams handles the complexity internally.

Decision Framework

Situation	Recommendation
Writing to database	Idempotent consumer
Calling external APIs	Idempotent consumer
Kafka Streams with state	exactly_once_v2
Multi-topic atomic writes	Transactions
Logging, metrics, analytics	At-least-once
"Duplicates are bad"	Idempotent consumer

The last row is intentional. "Duplicates are bad" isn't a reason to use EOS. It's a reason to make your consumers idempotent.

Stop reaching for exactly-once as the default. Start with at-least-once and idempotent consumers. Add EOS only when you've proven you need it and can afford the overhead.

Book a demo to see how Conduktor Console shows transaction states and helps identify when EOS overhead isn't paying off.