TAO keeps serving when a database or a cache fails, because a request has a second place to go.
- The leader DB is down. A follower is promoted to leader.
- A follower DB is down. Cache misses are sent to the leader DB.
- The leader fails. A read miss hits the DB. A write goes to a random leader in the leader tier, which writes to the DB, writes the inverse edge and sends invalidations to followers.
- A follower fails. Clients have two addresses for each shard, a primary follower and a backup follower. The Indians try Europe before the US.
The DB is the source of truth. A write fails only when the DB fails.
Refills and invalidations reach the followers through MySQL's binary log, which works like a message queue.
When the acknowledgement is not received, the leader queues the event to disk and tries later. The same write may be sent multiple times, so idempotency is expected.
The primary and the backup may not be perfectly consistent. Availability is more important.