How replication lag leaves a stale value in the cache

Cross-cluster consistency is keeping the cache of one region in step with a database that is written in another region.

Facebook runs clusters of Memcached in several regions, and every region holds all the data. One region holds the master databases and the other regions hold read only replicas. All writes go to the master because the system is read heavy: read queries are 100 times the write queries.

A write and a read in India race:

1. A web server in India writes K = 10 to the US database.
2. The same web server deletes K from the Indian cache.
3. Another Indian web server gets a read for K and misses in the cache.
4. It reads K = 5 from the Indian replica, which is the stale value.
5. The MySQL binlog replicates K = 10 to the Indian database.
6. The second web server sets K = 5 in the cache.

Every later read finds K = 5 in the cache, so the value never changes.