A remote marker is a Boolean flag that Memcached keeps on a deleted key. While it stands, Memcached does not apply a set for that key.
A set updates the database and deletes the entry from the cache. A get that misses reads the database and sets the cache. Across regions this races:
- A write updates the US database and deletes the key.
- A reader misses, picks the old value up from its database and sets it in the cache.
- The cache is stale till the next update.
With a remote marker, the delete tells Memcached that a remote update is coming. Memcached takes a set, gives a success and does not update the data. The system that reads the MySQL bin logs updates the Indian database and the cache.
There is only one remote marker and no counter. After two writes, a reader that comes once the marker is removed can still leave stale data.
The cache does not keep the updated value. It evicts it, which makes the rest of the system simpler.