What was cooking up interesting in the zoom class??
What's life without exploring? In this zoom class let us explore Kafka!
How many of us know about the Kafka internals?
Kafka is ubiquitous. The questions that come up to our mind when we want to dig deep into Kafka are:
How is Kafka Internally built?
It's a log. A log is a file where u write one line and when there is a new event you write another line. Every line will have a delimiter. Every line is appended after the delimiter.
Why Kafka doesn't work on queues(priority)?
Queues are implemented using linked lists or an array. The mechanism of Kafka doesn't give any priority. Kafka is fast, scalable, and distributed in nature by its design, partitioned and replicated commit log service. So there is no priority on topic or message.
All events belonging to a topic need to be ordered.
For example,
Let's say we have topics A B C and P Q R.
For instance, if all this is jumbled like P Q A B C R
Rules:
P comes before Q
And Q comes before R
The original order need not be maintained. We need to care about ordering between inside the topics not in between topics. Wherein C will be consumed by the consumer only after B gets consumed.
What is Causal ordering?
As long as relevant messages are sent to the same partition, message (causal) ordering is still guaranteed. In Kafka, this is achieved using keyed messages. Kafka partitioners would hash keys and ensure a key always goes to the same partition given the number of partitions is not changed.
How do u search in a log?
A log has a key and value. The log is already sorted by key. How we usually query is based on timestamp or the machine id. Instead of taking any random key, we take the 128 bits key and use 64 bits to store the timestamp.
For Example:
18th August_123
Where ‘_’ is used as a delimiter.
Elastic search
The basic idea is it is a no-sequel database. It has key-value pairs. It is good at text search and aggregations. You can add tags to it. An Elasticsearch index is a collection of documents that are related to each other.
Benefits of using Kafka
Limited persistence
Fast in terms of writing(if u want to read fast use another database).
Used for debugging
Finally, If a very popular search text exists how do u find the document that it exists in and store it efficiently?
I leave it here for you to discover. Let me give out a little secret!
This zoom class has the answer to this question!