Diskless Kafka: Direct-to-S3 Topics

Stéphane Derosiaux July 18, 2026 9 min read

Diskless Kafka is a topic type where brokers write incoming records directly to object storage such as Amazon S3 instead of to replicated local disk. Writes are leaderless (any broker can accept any partition), and a coordinator sequences the uploaded data and assigns offsets, moving durability off the brokers entirely.

diskless-kafka diagram 1

What KIP-1150 diskless topics are

KIP-1150 introduces diskless topics as an opt-in topic type that lives alongside classic topics in the same cluster. The design's own framing is "Diskless is to No-Disks as Serverless is to No-Servers": the disks do not disappear, but they stop being the durability layer for active data. Because it is a per-topic choice, operators can keep latency-sensitive workloads on classic disk-backed topics and move cost-sensitive, high-volume ones to diskless.

The mechanics live in follow-up KIPs that were still under discussion as of mid-2026:

KIP-1163 (Diskless Core): the write and read path
KIP-1164 (Diskless Coordinator): the batch coordinator
KIP-1165 (Object Compaction): merging small objects

How the write path works: leaderless writes and the batch coordinator

Classic Kafka assigns each partition a single leader broker; every produce request for that partition routes to the leader, which appends to its local log and replicates to follower brokers. Diskless removes the leader. Any broker can accept a produce request for any partition. That's a big change.

How does it work now?

A broker buffers incoming batches from many partitions, then uploads them together as a single shared log segment object (SLSO). Unlike a classic log segment (one partition per file) or a tiered-storage segment (one sealed partition segment), one SLSO holds data from many partitions written in the same time window.

This eliminates cross-broker replication for active data; durability is provided by the object store's own redundancy (S3 has 99.999999999% durability and stores every object across at least three AZs, can't beat it).

Ordering and offsets move to a "Diskless Coordinator", a new replicated-state-machine component that sequences uploaded batches, assigns the official per-partition offsets, and stores the batch-to-object metadata (for the read path to locate records).

A produce acknowledgement returns only after the batch is durable in object storage and the coordinator has assigned its offset. That two-step commit, with no local-disk fast path, is where the latency comes from with diskless topics.

Because small SLSOs create read amplification (a consumer of one partition would touch many objects), object compaction agents run in the background to reorder batches by offset, regroup them by topic-partition, and rewrite them into fewer, larger objects optimized for reads.

diskless-kafka diagram 2

Diskless vs tiered storage (KIP-405)

The two features are easy to confuse because both push data to S3.

The big difference is that Tiered storage keeps the active write path exactly as it is (records land on a leader's replicated local disk) and only offloads sealed, inactive segments to object storage after the fact. Diskless removes local-disk replication for active data by writing through to object storage on the produce path itself.

Dimension	Tiered storage (KIP-405)	Diskless (KIP-1150)
Status	GA since Kafka 3.9	Accepted March 2026; not in mainline
Write path	Leader broker → replicated local disk	Any broker → object storage directly
Partition leadership	Per-partition leader	Leaderless
What goes to S3	Sealed/inactive segments only	Active data, immediately
Cross-AZ replication	Still on the write path	Eliminated for diskless topics
Object layout	One partition per remote segment	Multi-partition Shared Log Segment
Ordering/offsets	Broker log	Batch Coordinator
Write latency	Unchanged (local disk)	Higher (object-store round trip)

A cluster can run classic, tiered, and diskless topics side by side, each chosen for its retention and latency profile.

Cost and latency trade-offs

The origin of diskless is to reduce cross-AZ replication traffic. Classic Kafka replicates each write across brokers in different AZs for durability, and vendors report that inter-AZ networking can reach 80–88% of cloud Kafka infrastructure cost (as of 2026, AWS and GCP both charge roughly $0.01/GiB per direction, ≈$0.02/GiB round-trip, for inter-zone transfer). Diskless routes durability through the object store, whose replication is internal and not separately billed, which cuts infrastructure cost.

Diskless adds a different cost: object stores bill per API request, so uploading many small SLSOs generates many PUTs, plus per-GB-month storage. Net savings therefore depend on throughput and object/compaction sizing: a low-volume topic writing tiny objects can see request charges eat much of the replication saving.

On Azure, inter-zone networking has historically been free, so diskless does not reduce the bill there. The savings come almost entirely from removing inter-AZ replication charges. See cross-AZ traffic in streaming.

The trade-off is latency. A produce that must land in object storage and be sequenced by the coordinator takes anywhere from ~100ms to 1–2s, versus a few milliseconds for classic disk-based topics. Both topic types can coexist in one cluster, this is a per-topic tuning decision, not a cluster-wide commitment.

The diskless ecosystem

KIP-1150 is an effort to standardize patterns vendors are already shipping:

Aiven Inkless: the open (AGPLv3) MVP fork implementing KIP-1150. Runs on Kafka 4.x, uses PostgreSQL as the coordinator (or an in-memory mode for testing), and supports S3, GCS, and Azure Blob (MinIO for local testing).
WarpStream: stateless agents, no local WAL, direct-to-S3, roughly 400–600ms P99; now part of Confluent/IBM.
AutoMQ: an Apache Kafka fork with a pluggable WAL: an S3 WAL (~500ms) or an EBS WAL (sub-10ms) for lower latency.
Confluent Freight: Using Kora's "direct write" leaderless mode; Confluent reports up to ~90% cost saving, latency up to 1–2s, no transactions.
Redpanda: markets Cloud Topics as a direct-to-object-storage write mode: payloads go to object storage on the produce path and keeps metadata and Raft consensus on local disk.

Note that when there is a coordinator in the data path, its availability is the data-plane SLA. Inkless depends on a database; WarpStream on a managed metadata plane. Tools like Conduktor add operational visibility across these varied backends, tracking produce/consume latency, offset lag, and topic-level configuration so teams can see how each backend behaves.

Is diskless Kafka available in Apache Kafka today?

No. KIP-1150 was accepted in March 2026 as a directional KIP, but it is not in mainline Apache Kafka. Stock releases have no diskless topic configuration. The implementation KIPs (1163/1164/1165) are still under discussion, and the only running implementation is the Aiven Inkless fork.

How is diskless Kafka different from tiered storage?

Tiered storage (KIP-405) keeps the active write path on replicated local disk and only offloads sealed segments to S3. Diskless removes disk replication for active data by writing directly to object storage on the produce path. They are complementary and can coexist per topic.

What is the batch coordinator?

A new replicated-state-machine component that sequences the batches brokers upload to object storage, assigns official per-partition offsets, and stores batch-to-object metadata for reads. It sits in the data path, so its availability directly affects the topic's SLA.

Why is diskless Kafka cheaper?

It eliminates cross-AZ replication traffic, which vendors report can be 80–88% of cloud Kafka infrastructure cost; durability moves to the object store's internal redundancy instead of inter-broker network transfer. Object-store API request and per-GB storage charges add some cost back, so net savings depend on throughput and object sizing. The saving is largest on AWS and GCP; on Azure, where inter-zone traffic is free, there is little benefit.

What is the latency cost of diskless topics?

Produce-to-consume latency rises to hundreds of milliseconds up to 1–2 seconds, versus sub-100ms for classic disk-based topics, because each write must reach object storage and be sequenced by the coordinator. KIP-1150 lets classic and diskless topics coexist, so latency-sensitive workloads stay on classic topics.

Tiered Storage in Kafka: the KIP-405 cold-tier offload that diskless complements rather than replaces.
Understanding KRaft Mode in Kafka: the metadata quorum model that coordinator-based designs build on.
Cross-AZ Traffic in Streaming: the inter-zone replication cost that diskless is designed to eliminate.
Streaming Total Cost of Ownership: how object-storage write paths reshape overall streaming cost.
Kafka Brokers Explained: the leader-based write path that leaderless diskless writes replace.
Kafka Replication and High Availability: the cross-broker replication diskless offloads to the object store.