Spark is one general framework for running computations, with libraries in place of the specialised MapReduce, ML, graph and interactive query engines.
A deployment scales to 10 thousand nodes. A SQL query is optimised before it runs: the filter is pushed down, so the filtering is done before the join.
Data is shared across computations in memory, and disk is the last resort:
- Interactive queries. A changed query runs on the old result in memory, a 100x speedup.
- Fault tolerance. If node 6 fails, nodes 1, 2 and 3 bring it back as node 7 from memory, in parallel, a 40x speedup.
- Streaming. Events are collected for 200 milliseconds into a micro batch and sent through the same pipeline. Merging the new result with the older result is left to the engineers.
An event pushed to Kafka goes through the pipeline already defined in Spark.
The operations are reduce, collect and foreach. Foreach is a for loop, collect gathers into a list, and reduce takes an average. Checkpoints for sorting and reduce operations happen in memory.