Spark runs the operations of a job together in memory, where MapReduce ran them stage by stage with a store and a transfer after every stage.
A common operation joins two tables on one key, such as a hundred GB of orders and 10 GB of profiles on the user profile key. In 2009 this was a big problem. The SQL query ran on the production DB for hours, or on a replica with instances spun up for it.
Google built MapReduce, and Apache Hadoop is a MapReduce framework:
- A job is a pipeline of stages.
- After every stage the results are stored in an intermediate data store and sent to the next node.
- The transfers take time, and the chance of a job crashing is proportional to the time it takes.
Spark puts operation one and operation two, a join and a filter on the same data, in one step. The engineer writes the steps as a script, and the framework finds the nodes.
MapReduce runs with less instance memory. Spark trades memory for speed.