What is a leader and follower in Kafka partitions?

In Apache Kafka, a partition is a fundamental unit of parallelism and replication. To ensure high availability, fault tolerance, and durability of data, each partition can have multiple replicas spread across different Kafka brokers. This replication mechanism operates on a leader-follower model, where one replica takes on the active role and others act as passive backups.

Kafka Partition Replication

For every partition, Kafka maintains a set of replicas. One of these replicas is designated as the 'leader', and the others are 'followers'. This architecture is crucial for Kafka's robustness, allowing it to continue operating even if some brokers fail.

The Leader

The leader is the primary replica for a given partition. It is responsible for handling all client read and write requests for that partition. When a producer sends a message to a topic, it's written only to the leader of the relevant partition. Similarly, when a consumer wants to read messages, it reads directly from the leader.

The leader also coordinates the replication process, ensuring that all its followers are keeping up with the data it has received. This centralized control prevents inconsistencies and simplifies client interactions.

The Follower

Followers are passive replicas that asynchronously pull new messages from the leader. Their primary role is to replicate the data stream from the leader to maintain an up-to-date copy of the partition's log. Followers do not serve any client requests directly.

The critical function of a follower comes into play during a leader failure. If the leader broker for a partition becomes unavailable (e.g., crashes or network issues), one of the healthy followers is automatically elected as the new leader. This failover process ensures continuous availability of the partition and minimal disruption to producers and consumers.

In-Sync Replicas (ISR)

Kafka defines an 'In-Sync Replicas' (ISR) set, which includes the leader and all followers that have fully caught up with the leader's log. Only replicas within the ISR are eligible to be elected as a new leader. This guarantees that no data loss occurs during a failover, as any newly elected leader will have a complete and consistent log.

Leadership Election

The process of electing a leader and managing the ISR set is handled by the Kafka controller, which historically relied on Apache ZooKeeper and now uses the Raft-based KRaft consensus protocol in newer Kafka versions. The controller monitors the health of brokers and initiates a leadership election when a leader fails.

In summary, the leader-follower model is central to Kafka's design, enabling its strong guarantees for data durability, fault tolerance, and high availability, making it a robust platform for real-time data streams.