How to count the frequency of words from 1TB of file in 2 GB RAM consistency having only a single machine?
There are 2 things that could be done in this case.
To split the whole file into contents of 2 GB.
The other approach would be to stream the file while tokenizing it and counting the number of words until it reaches the required size.
Data Structures that can be used:
Hash Maps
The entire words of the file will be converted into a key set of 1 Terabyte long if there are distinct words in the file.
The edge would be storing 1 next to all the hashed words of the file which will be altered with changing frequencies of the words.
We might need concurrent hashmaps to deal with this problem
The scatter-and-gather approach can be used to sort out the data.
Have a look at the external Merge Sort algorithm (Which can also be used to solve this problem)
Splitting the file into pieces is the complexity. Multi-threading can be used to solve this problem.
Let's say we have a fileSplitter(n),
There are 1,2,3 threads that are being passed. The splitter will split the file into pieces based on the number of words such that the limit memory is reached. We will have to stream through these chunks of threads. We will need to have seek() and parse() functions to go ahead with this method. Finally, we store all of this in hashmaps.
Quick Fact!!
BookMyShow makes use of Optimistic locks !! There are 2 copies to make sure your booking doesn't clash with already booked seats. While pessimistic locks won't create copies. The timestamps will help you to reach the point in the video where you can see a full video explanation of the same.
Why not solve your doubts?
When you are connecting to a server you are connecting to a port where multiple people connect. The server can identify who is connected to it. Let's go back to TCP connections,
We take a message and route it through multiple routers and send it to a person. It is more of a logical concept rather than a physical one.
Let's Move on to a system design problem:
There is an e-commerce app which has a lot of price fluctuations for its products. They have a catalogue of schools that is already in the ERP systems. They want to add a new component to the e-commerce platform. Get to know the tips on how to solve this!!
Come lets clear a user's doubt on over-utilizations of resources.