Apache Spark is a general purpose data analysis system that keeps intermediate results in memory.
The paper is from Berkeley. Before it, each system was specialised: Dremel for aggregation queries, MapReduce for batch jobs, Pregel for graph algorithms. Spark puts them under a single umbrella.
A MapReduce job runs on cheap commodity hardware that can crash:
- The restart. A crash restarts the entire process, which is slow.
- The disk. Intermediate results are stored and used during a failure. That takes disk space.
- The memory. Spark stores the results in memory. When a mapper fails, the output before it is resent from memory to a new mapper.
- The stages. No intermediate results are needed after every map job, so the functions run together in a single computer.
Spark focuses only on computation, so the cluster manager is pluggable. An operation is assigned to the computer that has its data in memory.