Kafka to Iceberg: Topics as Tables

Stéphane Derosiaux July 18, 2026 8 min read

Kafka to Iceberg is the pattern of exposing a Kafka topic as an Apache Iceberg table, making streaming records directly queryable by engines like Spark, Trino, and Snowflake.

There are two types of implementations:

topic-native approaches, where the broker materializes the table from the topic's own storage
copy-based sinks, where an external connector writes a separate Iceberg dataset.

kafka-to-iceberg-topics diagram 1

Why materialize topics as tables

A Kafka topic and a table are optimized for opposite access patterns.

A topic is a log: consumers read it sequentially, in offset order, streaming the whole thing from a position forward.
A table is built for selective/random access: a transactional row store answers point lookups and updates, while an analytical table (Iceberg over columnar Parquet) answers aggregations by reading only the columns and files a query touches, through partition pruning and predicate pushdown.

Exposing a topic as an Iceberg table is therefore a change of both paradigm and physical data layout, rewriting the same records from the log's sequential, record-at-a-time form into columnar files a query engine can scan selectively.

A Kafka topic's records can be exposed as rows in a table. The goal is that analysts and BI tools, which do not speak the Kafka protocol, can query the data in SQL over Iceberg through Spark, Trino, Flink, or Snowflake.

Historically the bridge was a simple integration with a connector: a consumer read the topic and wrote the records into an Iceberg table in object storage. It duplicates data and you have a connector to operate.

The newer approaches move that write into the broker directly: the bytes Kafka already retains become the Iceberg data files, so a topic surfaces as a queryable table with no external connector to run, and in some designs, without duplicating data at all.

Copy or Zero-Copy?

(A) Topic-native / broker-managed. The broker materializes the Iceberg table without a connector. Most builds extend tiered storage: segments landing in object storage are written as Parquet and committed to an Iceberg catalog (WarpStream and Confluent Cloud use their own object-storage materialization instead).

Two types of implementations:

Zero-copy: Aiven's RemoteStorageManager keeps a single Parquet object that serves both Kafka fetch and Iceberg queries (no second copy)
With copy: Redpanda and Confluent Tableflow keep the log and write distinct Iceberg files

(B) External copy-based sinks. A separate process like Kafka Connect or Flink consumes the topic and writes an independent Iceberg copy.

Dimension	(A) Topic-native / broker-managed	(B) Copy-based sink
Storage footprint	One shared object, no copy (Aiven RSM zero-copy); or a full second copy the broker writes (Redpanda, Confluent, WarpStream), same size as B	Always a full second copy of the topic
Freshness	Bound to the segment-roll or materialization cadence; no connector interval to tune	Bound to the connector commit interval (default 5 min); fresher means more small files
Operations	No connector to run; managed by the vendor, or self-hosted in the broker's storage plugin (Aiven)	You run and monitor Kafka Connect or Flink
Table maintenance	Still required, since streaming writes create small files; automated by managed vendors, but not by Aiven's self-hosted alpha	Yours to schedule: compaction, snapshot expiry, orphan-file cleanup
Lock-in / runs on	Requires a specific vendor's platform; the output is a standard Iceberg table any engine reads	Runs on any Kafka; open-source connector; the output is a standard Iceberg table too

Schema mapping: from Schema Registry to Iceberg schema

All methods derive the Iceberg schema from the Avro, Protobuf, or JSON schema managed in Schema Registry, mapping each field to an Iceberg column type. Inferring a schema from raw JSON is possible, but it often produces incorrect or suboptimal types (everything as string, numeric widths guessed), so a registry-backed subject is strongly preferred.

Schema evolution is handled in various ways:

Tableflow auto-evolves the table as the subject evolves (adding columns, widening types).
Connect evolves when iceberg.tables.evolve-schema-enabled is on.
Aiven's alpha build, by contrast, freezes the table on the first record's schema and does not yet handle evolution.

Compaction and table maintenance

Streaming's continuous writes create a housekeeping burden. Every commit interval or segment roll adds a small Parquet file, so a busy topic can produce thousands of tiny files an hour, and many small files slow reads: a query plans and opens across all of them, and per-file overhead dominates the scan.

Compaction must run regularly. It rewrites many small data files into fewer large ones (typically a few hundred MB each), which keeps scans fast and metadata small. It also folds the accumulated delete files into the data files so readers no longer reconcile deletes at query time.

This is one of several jobs every Iceberg table needs:

Snapshot expiry drops old snapshots so history and metadata do not grow without bound.
Orphan-file cleanup removes data files that no live snapshot references, left by failed or partial writes.
Manifest rewrites keep the manifest lists compact so query planning stays cheap.

Managed offerings do it for you. If you run the sink yourself, that maintenance is yours to schedule. See Maintaining Iceberg Tables and Iceberg catalog management for the deeper mechanics.

Does the topic-native path double my storage?

It depends on the design. Aiven's RemoteStorageManager stores a single Parquet object that serves both Kafka replay and Iceberg queries, so there is no duplication. Redpanda and Tableflow materialize broker-managed Iceberg files rather than a copy you operate. Copy-based sinks, by contrast, do store a full second copy you run and maintain.

Do I still need an Iceberg catalog?

Yes. Both families register the table in a catalog (REST, Glue, Nessie, Hive, or a vendor's open catalog). The catalog is what lets Spark, Trino, or Snowflake discover and read the table.

Is exactly-once guaranteed?

On the copy path, the Apache Iceberg Connect sink provides exactly-once via KIP-447 (Kafka 2.5+) plus a control topic for commit coordination. On the topic-native path it depends on the design: in the zero-copy approach (Aiven RSM) the analytical view inherits the broker's own durability because there is no separate copy to reconcile, while Redpanda and Tableflow reconcile a broker-managed materialization that the platform commits for you.

Can I backfill historical topic data into Iceberg?

It depends on the implementation. A copy-based sink can consume from the earliest offset. Redpanda, however, does not backfill an existing topic when you enable Iceberg; only records produced afterward appear in the table.

Which approach should I choose?

Choose topic-native for zero extra storage and no connector to operate when you are on a platform that offers it. Choose the copy-based Connect sink when you need an open, portable, engine-agnostic path across catalogs and are willing to run and maintain it.

Streaming to Lakehouse Tables: the broader pattern of landing streams as governed lakehouse tables.
Streaming Ingestion to Lakehouse: ingestion architectures that feed lake and warehouse engines.
Apache Iceberg: the open table format these approaches target.
Tiered Storage in Kafka: the KIP-405 mechanism the zero-copy builds extend.
Iceberg Catalog Management: Hive, Glue, and Nessie: how tables are registered and discovered.
Zero-Copy Data Sharing: the principle of one dataset serving multiple views.
Schema Registry and Schema Management: the source of the Iceberg schema in registry-driven paths.