The controller is one broker of a Kafka cluster, elected as the overall leader, which assigns the primary of every partition.
A single broker serves thousands of partitions, and Kafka has a set of brokers. One of them acts as the leader. ZooKeeper does the leader election for Kafka, through ZooKeeper atomic broadcast. The leader monitors the other brokers for failures.
Kafka uses ZooKeeper for cluster management:
- Registration. A broker registers with a unique ID.
- Notifications. Subscribers for registration and deregistration listen to /brokers/id.
- Removal. ZooKeeper has a snapshot of which brokers are alive. A broker that crashes does not respond to health checks on time, so ZooKeeper removes it from the cluster and no new messages reach it.
The controller chooses and reassigns the primary nodes for partitions. For partition P1 the leader is L1, and for partition P2 the leader is L2.
A topic T1 with 1000 partitions and three replicas for each partition is 3000 pieces of queue. Every partition needs a primary, so the controller selects 1000 primaries for this topic.