A gutter is a small set of servers that takes the requests of a failed cache server.
With S1, S2 and S3, a collapse of any one redirects its requests to S4, the gutter. Once S1 is back, its requests go back. This one backup reduced client failures by 99%.
The gutter is already working, so it is not autoscaling, and it is common to every server. It adds 1% more server capacity, where a backup for each doubles it.
The client's part:
- A get times out once. The client assumes the server has failed and queries the gutter.
- On a cache miss in the gutter, the client sets the entry. On a cache hit, it does nothing.
- Entries in the gutter expire sooner, so a profile changed in the gutter and in S1 is not inconsistent for long.
The load is not distributed over the other servers, because a single key can account for 20% of the requests to a server. A server that gets two or three of these keys is overloaded.