A consumer group makes sure that all the messages of one topic and partition are consumed by the same member of that group.
Without a group, two consumers take messages from the same queue. Debit 500 goes to consumer one, which takes 10 milliseconds, and credit 500 goes to consumer two, which takes one millisecond. The credit entry executes before the debit entry. With a very fast consumer, a transaction T2 executes before T1.
Inside a group the messages are accepted in order, distributed in order and processed in order. More consumers than topic and partition pairs are of no use.
Two consumers outside a group, C1 and C2, each have their own offset in Kafka, so both consume the same messages.
Kafka deals only with the offset. The guarantee is managed by the consumer:
- At most once. The consumer pulls 50 messages and increments its offset before processing them. If it collapses, those 50 are never processed.
- At least once. The consumer increments its offset only after it has processed the 50 messages.