A MapReduce job takes a big data set through many operations, stage by stage, on cheap computers.
Apache Spark is from 2009, Berkeley. Before it came MapReduce from Google: mapper jobs and reducer jobs.
The example is a recommendation on Amazon from four tables: orders, profiles, products and views. A user saw P1 the day before yesterday and P2 three months ago, and today there is a sale. The job gives each product a probability of purchase.
- The tables are loaded into different computers.
- Each is filtered by user name, date and location.
- The tables are joined. An IP address is mapped into a country.
- The output is an item wise probability list. The recommendation system takes the top 10.
A map operation converts one type of data into another. A reduce operation joins the tables and gets a single number.
So much data means things can fail, so the process is managed, and it runs on commodity hardware to stay cheap. Apache Spark is like the open source, in memory version of MapReduce.