A message is the unit of data in Kafka, like a row in a database, and the message key decides which partition holds it.
A message carries a body, a partition ID and a topic. A topic is like a queue, and a very large topic is split into pieces called partitions.
A message reaches its queue in two steps:
- The topic. The sender names it.
- The partition. The hash of the message key, modulo the number of partitions, gives the partition ID. The key can be the user ID of a profile, 356, or any data in the event. A custom router can replace this function.
Messages are ordered by topic and partition. One partition is processed first in, first out, never in parallel. Debit 500 from user A and credit 500 to user B, sent to the same topic and partition, are processed in that order, so the final state of the consumers is correct.
Partitions give scalability, reliability and ordering. A hot topic takes more partitions, and when one partition fails the others continue.