Gmail System Design

Chapter #1: Service Registration and Proxies

A gateway is the known place where external requests connect to the system, and it routes each request to the right internal service.

The domain name servers map gmail.com to the IP address of the gateway. Inside, the method auth in gmail.com/auth maps to the auth service.

The gateway also blacklists IP addresses that exceed a rate limit, and translates HTTP requests into a more efficient protocol over TCP.

The gateway now has a lot of responsibility, so the mapping moves to a service manager. Each service registers its method names and the IP addresses of its boxes, and a heartbeat tells the service manager which boxes are alive. The gateway caches the mapping.

1. The auth service stores the user ID, the encrypted password and the phone number, with verified set to false.
2. A code with a timestamp goes in a separate table, and the SMS service fills a template with it.
3. The user sends the code to gmail.com/verify. The auth service checks the code and its expiry and sets verified to true.

Chapter #2: Authentication & Global Caching

A token is a secret that the auth service gives a user at login and that the user sends with every subsequent request.

Every request to an internal service is authenticated. An unauthenticated request for the last day's emails could be sent by anyone who knows the user ID.

1. The username and password in a browser cookie, appended to every request. The password is stored as plain text.
2. A token in its place. Invalidating the token logs the user out. The gateway now asks the auth service on every request.
3. A cache of tokens at the gateway, updated by the auth service through a message queue. The gateway is bloated with somebody else's data, and an update fans out to every gateway box: three boxes, or 500.
4. A global cache, a map of key to value such as auth_userID to the secret. Services send updates to it, and the gateway makes a single API call to route and authenticate a request.

The global cache costs an API call, and the service stays decoupled.

Gmail Chapter #3: API contracts & Versioning

An API contract is the definition a service publishes of the objects its methods accept.

Creating a profile is a POST request that carries JSON. The gateway converts it into a programming object, which the service registered for that method name defines.

Each fix creates the next problem:

1. A contract. The service owns it and the gateway reads it to construct objects. Rules specific to the service stay out of the contract.
2. Internal services need the contract too, and new services keep arriving. It is stored where they register.
3. Pushing a contract change needs a restart of every box. Services pull the contract instead.
4. Contracts live in a contract registry. A framework like Thrift keeps them language independent.
5. A breaking change, an integer becoming a string, fails the parser. Contracts are versioned, and the sender chooses the version.

The profile image is uploaded to the drive service first and the request carries its URL. The user preference store maps a user ID to a layout and to the email preferences, both as JSON.

Chapter #4: Sending, Tagging & Searching Emails

Sending an email stores it in the email store and publishes an event to which the spam detector, the preference store and the search engine subscribe.

A send email request carries a JSON body to the email store. Its schema is an ID, to, from, subject and created timestamp. The content, with the text and the attachments, is kept separately as JSON.

1. Virus checker. It checks the URLs and uploads, and belongs to the drive service.
2. Spam detector. It reads the content and subject, and tags the email. 500 emails from the same sender with similar timestamps are spam. It runs as a cron job every hour, or as a stream service pulling email events from a message queue.
3. Preference store. It keeps a table of email ID to tags, which makes a search by tag fast.
4. Search engine. A join of the three tables on every search is slow, so Elasticsearch subscribes to the email event.

The subscribers are eventually consistent: a tag or spam mark takes a few seconds.

Gmail Chapter #5: Contacts & Groups

The contacts manager records who has emailed whom, and maps a group address to its members.

The email store publishes each email to a message queue such as Kafka. The spam detector, the preference store and the search engine subscribe, each at its own position.

Each addition creates the next problem:

1. Contacts. Sending an email makes the two users contacts, stored as a mapping of user ID from to user ID to. An email marked as spam creates no contact.
2. A second queue. It carries only emails whose tags have been set. The contacts manager and the search engine pull from it.
3. Groups. A group ID maps to user IDs, one row for each member of group 123.
4. External emails. Mail for another provider goes out through an SMTP server, and mail from outside is forwarded to the email store.

The final architecture is a gateway, an authentication service, a service registry, a preference store, the email store in three components, and these consumers.