# Kafka Tiered Storage: Infinite Retention Without Infinite Disks

Keeping 90 days of Kafka data costs 3x what it should. You're paying for fast SSDs to store data nobody reads.

Tiered storage fixes this by offloading old segments to object storage. Recent data stays on local disks; historical data moves to S3. Introduced in Kafka 3.0, it became GA in Kafka 3.6 with continued improvements in later versions.

> *We needed 7-year retention for SOX compliance. Traditional Kafka would've cost $600K/month. With tiered storage, we're at $58K. Same data, same durability.*
>
> *Data Architect at a financial services firm*

## The Cost Math

S3: $0.023/GB. EBS SSD: $0.08/GB. But Kafka replicates 3x. S3 handles durability internally, so you pay storage once (plus GET requests and data transfer for consumer reads). For EBS, you pay 3x.

| Storage Type | Per-GB Cost | After 3x Replication | 10 TB Monthly |
|--------------|-------------|---------------------|---------------|
| EBS SSD (gp3) | $0.08/GB | $0.24/GB | $2,400 |
| S3 Standard | $0.023/GB | $0.023/GB | $230 |

**Tradeoff:** Object storage is slower. Consumers reading historical data experience latency in seconds, not milliseconds. Tiered storage is for cold data, not your hot path.

## How It Works

Traditional Kafka stores everything on broker disks. With tiered storage:

- **Local tier:** Fast SSDs. Holds recent data (hours to days).
- **Remote tier:** S3/GCS/Azure. Holds historical data (weeks to years).

When a segment closes, the broker uploads it to remote storage. After `local.retention.ms`, the local copy is deleted.

## Enabling Tiered Storage

Kafka doesn't include a RemoteStorageManager. You need a plugin like [Aiven's tiered-storage-for-apache-kafka](https://github.com/Aiven-Open/tiered-storage-for-apache-kafka).

```properties
# server.properties
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.storage.manager.class.path=/opt/kafka/libs/tiered-storage/*

# S3 config
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3Storage
rsm.config.storage.s3.bucket.name=kafka-tiered-storage-prod
rsm.config.storage.s3.region=us-east-1

# Required for metadata manager
remote.log.metadata.manager.listener.name=PLAINTEXT
```

Then enable per topic (you can also configure these settings through a [topic management UI](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics)):

```bash
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic events-archive \
  --partitions 12 \
  --config remote.storage.enable=true \
  --config local.retention.ms=86400000 \
  --config retention.ms=7776000000
# local = 1 day on disk, total = 90 days including remote
```

## Two Retention Settings

| Setting | Applies To |
|---------|-----------|
| `local.retention.ms` | Time on local disk |
| `retention.ms` | Total retention (local + remote) |

Data flow: Producer writes → segment closes → uploads to remote → local deleted after `local.retention.ms` → remote deleted after `retention.ms`.

## Common Errors

**"Tiered Storage functionality is disabled"**

You enabled `remote.storage.enable=true` on a topic but the broker lacks `remote.log.storage.system.enable=true`.

**"No RemoteStorageManager configured"**

Kafka has no built-in S3 connector. Install a plugin and set `remote.log.storage.manager.class.name`.

**Consumer reads timing out**

Remote fetches take longer. Increase consumer timeouts:

```properties
fetch.max.wait.ms=5000
request.timeout.ms=60000
```

## Limitations

- Compacted topics not supported
- JBOD not supported
- No easy disable (must migrate and delete topic)
- Sequential remote fetches (parallel fetches planned for future releases)

## Recovery Time Improvement

This is where tiered storage shines:

| Operation | Without | With |
|-----------|---------|------|
| Disk failure recovery | 230 min | 2 min |
| Partition reassignment | 3 hours | 12 min |
| Cluster scale-up | 13+ hours | 1 hour |

New brokers don't copy terabytes of historical data. They only replicate local tier data; remote data is already durable.

For teams running multi-terabyte clusters with retention beyond a few days, tiered storage is worth the configuration effort.

[Book a demo](https://www.conduktor.io/contact/demo) to see how Conduktor Console provides unified visibility into local and remote storage metrics.