System Requirements
The system is WhatsApp, a chat-based application, and the requirements are the features chosen to fit an interview of one hour.
The special things about WhatsApp are group messaging and read receipts. The candidate features:
- Group messaging. A WhatsApp group holds at most 200 people.
- Image sharing, and video sharing.
- Sent, delivered and read receipts: the tick marks that show the stage a message is on.
- Whether the person is online, and the last time they were seen on the chat.
- Whether chats are temporary or permanent.
Temporary chats give the user privacy and power, and save storage space because the chats are stored only in the users' applications. Snapchat and WhatsApp are far more temporary than office messaging applications. If a user and the friend both delete the app, those chat messages are lost forever. Compliance and official communication need the message stored forever.
Image sharing is left out, which leaves four features. Group messaging is the first.
One to one messaging
One to one messaging is one person sending a message to another, with sent, delivered and read receipts going back to the sender.
A user to box mapping stored in the gateway takes memory away from TCP connections and is duplicated on every box. So the connection is kept dumb, and a sessions microservice stores who is connected to which box.
- A sends the message for B to gateway 1, which passes it to the sessions service.
- The message is stored in a database and retried till B gets it. A gets the sent receipt.
- The sessions service finds B's box and routes the message to gateway 2, which sends it to B.
- B acknowledges to gateway 2. The sessions service finds A's box, and A gets the delivery receipt.
- When B opens the chat, the same flow carries the read receipt.
HTTP is a client to server protocol, so the server cannot send a message to B. Long polling is not real-time. WebSockets over TCP let the server send to the client.
Last Seen Timestamps
The last seen timestamp is the time of a user's latest activity, stored on the server as a key value pair of user to timestamp.
B wants to know when A was last online. B asks the server, and A is not in the picture.
- Every activity of A, such as sending or reading a message, is a request to the server, and the current timestamp is persisted in the table.
- If A was online three seconds ago, B is shown online. The threshold is 10 or 15 seconds.
- A last seen microservice behind the gateway does the user activity tracking.
- Some requests are sent by the application itself, such as polling for messages or a delivery receipt. The client flags each request as a user activity or an app request.
- Only user activities go to the last seen service, which updates the timestamp for that user.
B queries this service and is told that the user is online, or was last seen at the timestamp.
Group Messaging
Group messaging sends one user's message to every other member of the group.
Who is in which group is decoupled from the session service into a group service.
- A user sends a group message to a gateway, which forwards it unparsed.
- A parser microservice converts it to an object in the internal protocol, such as Thrift.
- The session service asks the group service for the other members of the group, and gets their user IDs.
- The session service finds the gateway each user is connected to, and routes the message to each one.
A group is limited in size, 200 on WhatsApp, because a larger group fans out the request too much.
Requests to the group service are routed on the group ID with consistent hashing. A message queue retries a failed request, and after five retries the failure is propagated to the client.
Delivered and seen receipts for a group are expensive. Under heavy load such as New Year's, unimportant messages like last seen are deprioritized.