Kafka topic configuration: log retention

Kafka retention policy explained: set Kafka topic retention with retention.ms and log.retention.bytes, choose time-based vs size-based retention, plan storage.

Learn how to configure data retention policies in Kafka

Kafka log retention controls how long messages are stored before being deleted. Understanding retention configuration is essential for managing storage costs, compliance requirements, and consumer catch-up scenarios.

What you'll learn:

  • How Kafka's delete cleanup policy works
  • How to configure retention by time and size
  • The relationship between broker and topic-level settings
  • Common retention patterns for different use cases

Log retention overview

Kafka stores messages for a set amount of time and purges messages older than the retention period. This expiration happens due to the log.cleanup.policy=delete policy (the default for user topics).

Within a topic partition, segments past the retention time or size are deleted while segments within retention are kept, and the active segment is always kept

Retention configuration options

Retention by time

The most common configuration for how long Kafka will retain messages is by time.

SettingDefaultDescription
log.retention.hours168 (7 days)Retention time in hours
log.retention.minutes-Retention time in minutes
log.retention.ms-Retention time in milliseconds
If more than one is specified, the smaller unit size will take precedence.

Use milliseconds for consistency

Because the Kafka CLI command only allows you to set the ms version of this parameter, we recommend using retention.ms across all your configurations.

Retention by size

Another way to expire messages is based on the total number of bytes of messages retained.

SettingDefaultDescription
log.retention.bytes-1 (unlimited)Maximum size per partition
The default is -1, meaning that there is no limit and only a time limit is applied. This parameter is useful to set a to positive value if you want to keep the size of a log under a threshold.

Broker-level vs Topic-level

Kafka broker-level topic configurations are prefixed by log. and we can remove it to find the equivalent Kafka topic-level configuration. For example, log.retention.ms becomes retention.ms at the topic level.

How retention is applied

Diagram illustrating the process Kafka uses to delete old log segments based on the time or size rules configured for the Kafka Topic.

Retention by time is performed by examining the last modified time on each log segment file on disk. This is the time that the log segment was closed, and represents the timestamp of the last message in the file.

If you have specified a value for both log.retention.bytes and log.retention.hours, messages may be removed when either criteria is met.

Minimum guarantees, not hard limits

These are minimum guarantees, not hard limits. The active segment does not count toward the byte limit, and the time limit can be much greater than expected if the segment is very big (few messages per day in a 1GB segment).

Common retention patterns

One week of retention (default)

retention.ms=604800000    # 7 days in milliseconds
retention.bytes=-1        # No size limit

Infinite time retention bounded by 500MB

retention.ms=-1           # No time limit
retention.bytes=524288000 # 500MB per partition

Short retention for high-volume topics

retention.ms=86400000     # 24 hours
retention.bytes=-1        # No size limit

Compliance: 90 days retention

retention.ms=7776000000   # 90 days
retention.bytes=-1        # No size limit

Configure retention

To set these configurations using the CLI:

kafka-configs --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name configured-topic \
  --add-config retention.ms=-1,retention.bytes=524288000

Retention decision guide

Decision tree for retention: set retention.ms for compliance, else retention.bytes for storage limits, else a longer retention.ms for historical data or a shorter one otherwise, always monitoring disk usage

See it in practice with Conduktor

Conduktor Console displays topic retention settings and current storage usage. Monitor disk space consumption to validate your retention configuration meets both storage and compliance needs.

The Insights dashboard identifies empty, stale, and tiny topics that may be consuming unnecessary storage, helping you make data-driven decisions about retention policies and topic cleanup.

Next steps