# Kafka topic configuration: log retention

*Learn how to configure data retention policies in Kafka*

Kafka log retention controls how long messages are stored before being deleted. Understanding retention configuration is essential for managing storage costs, compliance requirements, and consumer catch-up scenarios.

**What you'll learn:**
- How Kafka's delete cleanup policy works
- How to configure retention by time and size
- The relationship between broker and topic-level settings
- Common retention patterns for different use cases

## Log retention overview

Kafka stores messages for a set amount of time and purges messages older than the retention period. This expiration happens due to the `log.cleanup.policy=delete` policy (the default for user topics).

![Within a topic partition, segments past the retention time or size are deleted while segments within retention are kept, and the active segment is always kept](https://www.conduktor.io/assets/kafka/diagrams/kafka-topic-configuration-log-retention--1.svg)

```mermaid
flowchart LR
    subgraph Topic["Topic Partition"]
        S1["Segment 1<br/>Oldest"]
        S2["Segment 2"]
        S3["Segment 3"]
        S4["Active Segment<br/>Newest"]
    end

    S1 -->|"Past retention<br/>time/size"| Delete["Delete"]
    S2 -->|"Within retention"| Keep["Keep"]
    S3 --> Keep
    S4 -->|"Always kept"| Keep
```

## Retention configuration options

### Retention by time

The most common configuration for how long Kafka will retain messages is **by time**.

| Setting | Default | Description |
|---------|---------|-------------|
| `log.retention.hours` | 168 (7 days) | Retention time in hours |
| `log.retention.minutes` | - | Retention time in minutes |
| `log.retention.ms` | - | Retention time in milliseconds |

If more than one is specified, the smaller unit size will take precedence.

> **Use milliseconds for consistency**
> Because the Kafka CLI command only allows you to set the `ms` version of this parameter, we recommend using `retention.ms` across all your configurations.

### Retention by size

Another way to expire messages is based on the total number of bytes of messages retained.

| Setting | Default | Description |
|---------|---------|-------------|
| `log.retention.bytes` | -1 (unlimited) | Maximum size per partition |

The default is **-1**, meaning that there is no limit and only a time limit is applied. This parameter is useful to set a to positive value if you want to keep the size of a log under a threshold.

> **Broker-level vs Topic-level**
> Kafka broker-level topic configurations are prefixed by `log.` and we can remove it to find the equivalent Kafka topic-level configuration. For example, `log.retention.ms` becomes `retention.ms` at the topic level.

## How retention is applied

![Diagram illustrating the process Kafka uses to delete old log segments based on the time or size rules configured for the Kafka Topic.](https://www.conduktor.io/assets/kafka/Adv-Kafka-Topic-Log-Comp-1.png)

Retention by time is performed by examining the last modified time on each log segment file on disk. This is the time that the log segment was closed, and represents the timestamp of the last message in the file.

If you have specified a value for both `log.retention.bytes` and `log.retention.hours`, messages may be removed when either criteria is met.

> **Minimum guarantees, not hard limits**
> These are minimum guarantees, not hard limits. The active segment does not count toward the byte limit, and the time limit can be much greater than expected if the segment is very big (few messages per day in a 1GB segment).

## Common retention patterns

### One week of retention (default)

```properties
retention.ms=604800000    # 7 days in milliseconds
retention.bytes=-1        # No size limit
```

### Infinite time retention bounded by 500MB

```properties
retention.ms=-1           # No time limit
retention.bytes=524288000 # 500MB per partition
```

### Short retention for high-volume topics

```properties
retention.ms=86400000     # 24 hours
retention.bytes=-1        # No size limit
```

### Compliance: 90 days retention

```properties
retention.ms=7776000000   # 90 days
retention.bytes=-1        # No size limit
```

## Configure retention

To set these configurations using the CLI:

```bash
kafka-configs --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name configured-topic \
  --add-config retention.ms=-1,retention.bytes=524288000
```

## Retention decision guide

![Decision tree for retention: set retention.ms for compliance, else retention.bytes for storage limits, else a longer retention.ms for historical data or a shorter one otherwise, always monitoring disk usage](https://www.conduktor.io/assets/kafka/diagrams/kafka-topic-configuration-log-retention--2.svg)

```mermaid
flowchart TD
    Start["Configure retention"] --> Q1{"Compliance<br/>requirements?"}

    Q1 -->|"Yes"| Compliance["Set retention.ms based<br/>on compliance period"]
    Q1 -->|"No"| Q2{"Storage<br/>constraints?"}

    Q2 -->|"Yes"| Size["Set retention.bytes<br/>to limit size"]
    Q2 -->|"No"| Q3{"Need historical<br/>data?"}

    Q3 -->|"Yes"| Long["retention.ms = longer<br/>(30+ days)"]
    Q3 -->|"No"| Short["retention.ms = shorter<br/>(1-7 days)"]

    Compliance --> Monitor["Monitor disk usage"]
    Size --> Monitor
    Long --> Monitor
    Short --> Monitor
```

> **See it in practice with Conduktor**
> [Conduktor Console](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) displays topic retention settings and current storage usage. Monitor disk space consumption to validate your retention configuration meets both storage and compliance needs.
> The [Insights dashboard](https://docs.conduktor.io/guide/insights/cost-control) identifies empty, stale, and tiny topics that may be consuming unnecessary storage, helping you make data-driven decisions about retention policies and topic cleanup.

## Next steps

- [Understand log compaction](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction) as an alternative cleanup policy
- [Learn about log segments](https://www.conduktor.io/kafka/kafka-topics-internals-segments-and-indexes) for deeper understanding
- [Configure topics with CLI](https://www.conduktor.io/kafka/how-to-change-a-kafka-topic-configuration-using-the-cli) for more options