Kafka Exactly-Once: When It Works and When It Doesn't

"We enabled exactly-once but we're still seeing duplicates in the database."

It is a common complaint. Teams enable Kafka's EOS settings, assume the problem is solved, then discover duplicates in their downstream systems. The mistake is conceptual: exactly-once stops at the edge of Kafka, and their database is outside it.

Kafka's exactly-once guarantees atomicity for reads and writes within the cluster. Everything beyond that boundary is your responsibility.

What Idempotent Producers Actually Solve

An idempotent producer ensures retrying a failed send never creates duplicates in the topic. Each producer gets a unique ID. Every batch includes a sequence number. The broker deduplicates based on this pair.

props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");

Since Kafka 3.0, idempotence and acks=all are both defaults. Idempotence stays on unless you set a conflicting config such as acks=1, in which case it is silently disabled. With it on, retries are safe: no broker-level duplicates.

The limitation: The Producer ID doesn't survive restarts. If your producer crashes and restarts, it gets a new ID. The broker can't detect duplicates from the old instance.

Producer Instance 1 (PID=100): Send message A → Written
                               [CRASH]
Producer Instance 2 (PID=101): Send message A → Written again

Idempotent producers protect against network-level duplicates. They don't protect against application-level retries after a restart.

Transactions: Atomic Writes Across Partitions

Transactions survive restarts through a persistent transactional.id: a new instance with the same ID aborts the old instance's unfinished transaction and fences it out. They don't recognize a record your application sends again after a transaction already committed. That only goes away when the input offsets commit in the same transaction (next section).

props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-processor-1");
producer.initTransactions();

producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", key, "order-created"));
producer.send(new ProducerRecord<>("inventory", key, "stock-decremented"));
producer.commitTransaction();

On commit, all messages become atomically visible. On abort, consumers with read_committed never see them.

The transactional.id enables "zombie fencing": if two producers use the same ID, the older one gets fenced out. This prevents split-brain scenarios.

The Consume-Transform-Produce Pattern

The canonical exactly-once use case: read from input, transform, write to output, commit offsets, all atomically.

consumerProps.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
producerProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "processor-1");

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    producer.beginTransaction();

    for (ConsumerRecord<String, String> record : records) {
        producer.send(new ProducerRecord<>("output", record.key(), transform(record.value())));
    }

    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
    producer.commitTransaction();
}

If commitTransaction() succeeds, output messages are visible and consumer offsets are committed. If it fails, everything rolls back. On restart, you reprocess from the last committed offset.

Kafka Streams makes this trivial:

props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

Where Exactly-Once Does NOT Work

This is where most teams get burned.

External Systems

Kafka transactions can't include your database or REST API:

producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", key, order));
restClient.post("https://payments.example.com/charge", order);  // NOT TRANSACTIONAL
producer.commitTransaction();

If the REST call succeeds but commitTransaction() fails, the payment went through but the Kafka message rolled back. You now have inconsistency.

Solution: Write to Kafka only. Propagate to external systems with idempotent consumers.

Cross-Cluster Replication

Transactions don't span clusters. MirrorMaker 2 supports exactly-once replication since Kafka 3.5 (dedicated mode, exactly.once.source.support=enabled), but that covers the copy only. Consumers that fail over resume from translated offsets, which lag the source cluster, so they reprocess some records.

Sink Connectors

Kafka → PostgreSQL via JDBC Sink is at-least-once. The database is outside Kafka's transaction boundary.

Mitigation: Use idempotent writes. The JDBC sink can upsert using primary keys instead of insert.

Side Effects

builder.stream("orders")
    .foreach((key, value) -> sendEmailNotification(value));  // NOT exactly-once

If the stream task fails after sending the email but before committing, the email sends again on restart.

Solution: Write to an output topic. Consume with a separate service that tracks sent emails.

Performance Tradeoffs

Confluent's original benchmark (2017, Kafka 0.11) found idempotence had negligible throughput impact, and a transactional producer lost 3% of throughput against acks=all with 1 KB records and 100 ms transactions. Kafka Streams EOS with a 100 ms commit interval lost 15 to 30%, and nothing measurable with a 30-second interval and 1 KB records (source). Measure your own workload.

The overhead is per transaction, so for high-throughput applications, batch more messages per transaction. The tradeoff is latency: read_committed consumers only see records once their transaction commits.

When to Use What

Scenario	Recommendation
Logging, metrics	At-least-once
General pipelines	Idempotent producer
Kafka → Kafka processing	Transactions or Streams EOS
Kafka → Database	At-least-once + idempotent sink

The hardest part of exactly-once isn't enabling it. It's understanding where it stops. Kafka guarantees atomicity within the cluster. Everything beyond requires idempotent design on your side.

Book a demo to see how Conduktor Console shows transaction markers and consumer states across your clusters.