Cold cluster warmup fills a new cluster's cache from a warm cluster's memcached, which reduces the time to heat up from days to hours.
A new cache cluster is completely empty. A get request misses, the value is set, and the next get request misses again, so the cache is useless until it is populated.
On a miss the cold cluster does not hit the database. It hits a warm cluster's memcached, a cross region call, because something popular in one cluster is likely popular in another. Mcrouter does this based on a config flag, until the new cluster has normal hit rates.
This creates a race condition:
- Initially K is 5.
- The cold cluster sets K to 10 in the database.
- A get for K misses in the cold cluster and hits the warm cluster, which returns 5.
- Mcsqueal invalidates K in the warm cluster.
The cold cluster holds 5 and the database holds 10. The database never tells the cold cluster to reset the value, so it stays stale.