Kafka writes messages in batches: a producer sends up to 50 events in one call, in place of one call and one acknowledgement for each.
A consumer batches too. It names its offset and asks for 50 messages. Kafka sends 50, or all it has when there are fewer.
Batching trades latency for throughput. Messages wait in memory, so the system is no longer real time, and throughput increases.
Three things follow for a batch:
- A batch goes to one topic and partition. A producer that writes to three queues holds a batch for each, which consumes memory. The client can force a send of fewer than 50.
- A batch is a single API call, and its messages are added to the queue in order.
- A client that collapses loses its batch, unless it stored the messages.
The producer and the consumer batch at their own rates, because the messages are stored in between.
A batch is compressed: the metadata of 50 messages is reduced. Messages can have schemas in Avro, a format like protobuf.