Kafka Exactly-Once: When It Works and When It Doesn't

Stéphane Derosiaux October 15, 2025 4 min read
Kafka Exactly-Once: When It Works and When It Doesn't

"We enabled exactly-once but we're still seeing duplicates in the database."

It is a common complaint. Teams enable Kafka's EOS settings, assume the problem is solved, then discover duplicates in their downstream systems. The mistake is conceptual: exactly-once stops at the edge of Kafka, and their database is outside it.

Kafka's exactly-once guarantees atomicity for reads and writes within the cluster. Everything beyond that boundary is your responsibility.

What does this post change for your team?ClaudeChatGPT

What Idempotent Producers Actually Solve

An idempotent producer ensures retrying a failed send never creates duplicates in the topic. Each producer gets a unique ID. Every batch includes a sequence number. The broker deduplicates based on this pair.

props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");

Since Kafka 3.0, idempotence and acks=all are both defaults. Idempotence stays on unless you set a conflicting config such as acks=1, in which case it is silently disabled. With it on, retries are safe: no broker-level duplicates.

The limitation: The Producer ID doesn't survive restarts. If your producer crashes and restarts, it gets a new ID. The broker can't detect duplicates from the old instance.

Producer Instance 1 (PID=100): Send message A → Written
                               [CRASH]
Producer Instance 2 (PID=101): Send message A → Written again

Idempotent producers protect against network-level duplicates. They don't protect against application-level retries after a restart.

Transactions: Atomic Writes Across Partitions

Transactions survive restarts through a persistent transactional.id: a new instance with the same ID aborts the old instance's unfinished transaction and fences it out. They don't recognize a record your application sends again after a transaction already committed. That only goes away when the input offsets commit in the same transaction (next section).

props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-processor-1");
producer.initTransactions();

producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", key, "order-created"));
producer.send(new ProducerRecord<>("inventory", key, "stock-decremented"));
producer.commitTransaction();

On commit, all messages become atomically visible. On abort, consumers with read_committed never see them.

The transactional.id enables "zombie fencing": if two producers use the same ID, the older one gets fenced out. This prevents split-brain scenarios.

The Consume-Transform-Produce Pattern

The canonical exactly-once use case: read from input, transform, write to output, commit offsets, all atomically.

consumerProps.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
producerProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "processor-1");

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    producer.beginTransaction();

    for (ConsumerRecord<String, String> record : records) {
        producer.send(new ProducerRecord<>("output", record.key(), transform(record.value())));
    }

    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
    producer.commitTransaction();
}

If commitTransaction() succeeds, output messages are visible and consumer offsets are committed. If it fails, everything rolls back. On restart, you reprocess from the last committed offset.

Kafka Streams makes this trivial:

props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

Where Exactly-Once Does NOT Work

Most teams get burned here.

External Systems

Kafka transactions can't include your database or REST API:

producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", key, order));
restClient.post("https://payments.example.com/charge", order);  // NOT TRANSACTIONAL
producer.commitTransaction();

If the REST call succeeds but commitTransaction() fails, the payment went through but the Kafka message rolled back. You now have inconsistency.

Solution: Write to Kafka only. Propagate to external systems with idempotent consumers.

Cross-Cluster Replication

Transactions don't span clusters. MirrorMaker 2 supports exactly-once replication since Kafka 3.5 (dedicated mode, exactly.once.source.support=enabled), but that covers the copy only. Consumers that fail over resume from translated offsets, which lag the source cluster, so they reprocess some records.

Sink Connectors

Kafka → PostgreSQL via JDBC Sink is at-least-once. The database is outside Kafka's transaction boundary.

Mitigation: Use idempotent writes. The JDBC sink can upsert using primary keys instead of insert.

Side Effects

builder.stream("orders")
    .foreach((key, value) -> sendEmailNotification(value));  // NOT exactly-once

If the stream task fails after sending the email but before committing, the email sends again on restart.

Solution: Write to an output topic. Consume with a separate service that tracks sent emails.

Performance Tradeoffs

Confluent's original benchmark (2017, Kafka 0.11) found idempotence had negligible throughput impact, and a transactional producer lost 3% of throughput against acks=all with 1 KB records and 100 ms transactions. Kafka Streams EOS with a 100 ms commit interval lost 15 to 30%, and nothing measurable with a 30-second interval and 1 KB records (source). Measure your own workload.

The overhead is per transaction, so for high-throughput applications, batch more messages per transaction. The tradeoff is latency: read_committed consumers only see records once their transaction commits.

When to Use What

ScenarioRecommendation
Logging, metricsAt-least-once
General pipelinesIdempotent producer
Kafka → Kafka processingTransactions or Streams EOS
Kafka → DatabaseAt-least-once + idempotent sink
The hardest part of exactly-once isn't enabling it. It's understanding where it stops. Kafka guarantees atomicity within the cluster. Everything beyond requires idempotent design on your side.

Book a demo to see how Conduktor Console shows transaction markers and consumer states across your clusters.