TAO keeps a copy of its caches and databases in every region, and sends every write to one of them.
One deployment in the US makes each like from India a call across continents, so the data is replicated around the globe. A social graph cannot be split by location, because a person in Hyderabad has relatives in the US. A write made in India has to reach the US too.
- The log. Every create, update and delete is a line in a log. Replaying the lines rebuilds the database, which makes a replica.
- The exchange. Regions that all take writes exchange logs and replay the lines each is missing. In MySQL this log is the bin log.
- The cost. When every region is a peer, each database spends its time on updates from the others, and throughput goes down.
- The leader. Of a thousand operations, two are writes. All writes go to one region and every region serves reads.
Clients always connect with the closest follower, never directly with the leader.