Kafka is a system that accepts events, stores them in a persistent store and distributes them to all interested parties.
Around 2011, large scale internet systems like Google, Facebook and LinkedIn had problems consuming so much data and storing it. Kafka was developed by LinkedIn.
A celebrity posts on LinkedIn. That single message has to reach many notification servers. Kafka sits in between:
- The producer sends the message. Kafka says it got the message and takes care of the delivery, at least once or at most once.
- Kafka keeps the message for a retention period of one or two weeks and retries it. A subscriber that does not accept it in that time loses it.
- Publishers only publish and consumers only consume. Kafka scales up as the message rates increase.
The architecture becomes three parts: producers, distributors and consumers.
Kafka was made for the producer consumer architecture, retention of messages on disk, scale (petabytes of data per day in 2015, recently something like 7 trillion messages per day), high performance, simplicity for its clients and reliability.