A broker is a Kafka server. It receives messages from producers, assigns each one an offset and commits it to disk, like a log. A consumer mentions its offset and the broker reads from there.
A single broker serves thousands of partitions. When one server goes down, only some partitions of a topic are affected, and the pieces of a queue are consumed in parallel.
Each partition is replicated into multiple brokers for redundancy:
- One copy is the primary, the others are replicas. A replica can be one message behind, so consumers read only from the primary.
- If the primary fails, a replica that is in sync is promoted: one that pulled all the messages in the last 10 seconds.
- A replica that is out of sync cannot stand in the election, but it can vote.
The replicas are not read. They are for fault tolerance.
Retention is set per topic by time or by storage. When it runs out the oldest message is dropped, and only one member of a consumer group is affected.