A cascading failure is a collapse in which the load of one failed server brings down the servers that remain.
When a server collapsed at Facebook, it took time to find out, and then scripts had to be run to start it again. In 2011 FBAR took the common parts of the fixes engineers wrote for their failed scripts and made them an API. Even with this automation, a failure takes a few minutes to be detected and fixed.
In those minutes the load on the other servers increases:
- Three servers, S1, S2 and S3, each run at 60% capacity behind a load balancer.
- S1 fails. Its 60 units of work are distributed equally, so S2 and S3 each carry 30 plus 60 out of 100.
- At 90% the chance of a server failing is high. When S2 fails, all the load hits S3.
Facebook's answer is the gutter, a separate set of servers, S4. All of S1's requests are routed to the gutter before S2 can fail.