Spark recovers from a crashed node by asking the nodes it depended on for their results, which are kept in memory.
MapReduce keeps intermediate results, so a failure in between is not a problem. Spark keeps most things in memory, and intermediate results would lose that purpose.
The job is a directed acyclic graph of nodes:
- Node 6 crashes. It depended on nodes 1, 2 and 3.
- A new node 7 is spun up. Nodes 1, 2 and 3 give it their results from memory.
- The data node 6 held is lost forever, and node 7 computes it again.
- With multiple failures this becomes slow.
A checkpoint pushes data to a disk, and one after every line of code takes a lot of bandwidth. One checkpoint at the end is a single movement to disk, but data lost at the last line means performing all the operations again.
Hadoop made sense for jobs hours long, because its intermediate results save recomputing after machine failures. Spark has taken those benefits over.