What is the role of a Kafka broker?

A Kafka broker is a fundamental component of the Apache Kafka distributed streaming platform. It acts as a server that stores and manages messages, forming the backbone of a Kafka cluster.

Overview

In Apache Kafka, a broker is essentially a server (or node) in the Kafka cluster responsible for storing data (messages) for one or more topics. It handles all requests from producers, consumers, and other brokers, ensuring the reliable and scalable operation of the distributed streaming system.

Core Responsibilities

• Message Storage and Persistence: Brokers store messages in log-structured files on disk. Each topic is divided into partitions, and these partitions are spread across different brokers.
• Message Replication: To ensure high availability and fault tolerance, partitions are replicated across multiple brokers. One broker acts as the 'leader' for a partition, while others are 'followers', maintaining synchronized copies of the data.
• Serving Producer and Consumer Requests: Brokers accept messages from producers, append them to the appropriate topic partitions, and serve messages to consumers based on their offset, managing read and write requests.
• Metadata Management: Brokers maintain metadata about the cluster, including information about topics, partitions, and their leaders/followers, which is crucial for client interactions and cluster health.
• Cluster Coordination: Brokers coordinate with each other and the cluster controller (historically Zookeeper, now often Kraft) to manage partition assignments, handle leader elections, and ensure overall cluster stability and health.

How Brokers Work Together

A Kafka cluster consists of one or more brokers. For any given topic, its partitions are distributed among these brokers. Each partition has a designated leader broker and one or more follower brokers. If the leader fails, one of the followers is elected as the new leader, ensuring continuous data availability and fault tolerance. This distributed architecture allows Kafka to scale horizontally, handling massive amounts of data and high throughput efficiently.