What is message retention in Kafka?

Message retention in Apache Kafka refers to the policy that dictates how long messages are stored on the Kafka brokers before they are permanently deleted. Unlike traditional message queues that remove messages immediately after consumption, Kafka retains messages for a configurable period, making them available for multiple consumers or for re-processing.

What is Message Retention?

Kafka's message retention is a fundamental feature that enables its robust, distributed, and fault-tolerant architecture. It ensures that messages written to a topic partition remain available for a specified duration, regardless of whether they have been consumed or not. This design choice provides flexibility and resilience, allowing consumers to process data at their own pace and even re-read historical data if needed.

How Kafka Manages Retention

Kafka stores messages in immutable, append-only log segments on disk. Message retention is managed at the partition level. When a message is produced, it's appended to the current active log segment. Once a log segment reaches a certain size or time threshold, it's rolled over, and a new active segment is created. Retention policies then apply to these completed log segments, not individual messages.

Retention Policies

Kafka primarily uses two types of retention policies, which can be configured at the broker or topic level:

• Time-based (log.retention.hours or log.retention.ms): This policy defines the maximum time a log segment will be retained. Once a segment's age (based on its creation timestamp) exceeds this configured duration, it becomes eligible for deletion. The default is typically 7 days (168 hours or 604,800,000 ms).
• Size-based (log.retention.bytes): This policy defines the maximum size a partition's log can grow to. If the total size of all log segments for a partition exceeds this value, the oldest segments are deleted until the partition's log size falls below the limit. The default is often -1, meaning no size limit.

Kafka continuously checks for eligible log segments to delete based on these policies. When both time and size retention policies are set, the first one that triggers deletion will be applied.

Key Benefits

• Fault Tolerance & Replayability: Consumers can crash and restart, or new consumers can join, and still process all historical messages within the retention period.
• Decoupled Consumers: Producers and consumers are fully decoupled. Producers don't need to know about consumers, and consumers can operate at their own pace without affecting message availability for others.
• Stream Processing Flexibility: Enables various stream processing patterns, allowing applications to re-process data, build different views of the same data, or backfill new analytical models.

Configuration

Retention settings can be configured globally at the broker level (applying to all topics unless overridden) or overridden for specific topics. Topic-level configuration provides fine-grained control for different data streams.

kafka-configs.sh --bootstrap-server localhost:9092 --alter --topic my_topic --add-config retention.ms=86400000
kafka-configs.sh --bootstrap-server localhost:9092 --alter --topic my_topic --add-config retention.bytes=1073741824