Converting data sources to a lake
A data warehouse is a place where the data generated by business services is collected and used later: analytics, alerts and monitoring, a query dashboard.
In a food delivery system the profile, sessions, trips, restaurants and payments databases need not match. Reading from them for every insight puts load on databases the main services need, so a copy of all of them is made into a data lake.
The lake is heterogeneous. Its records are linked by something common across the databases: the delivery ID. One ID links five types of data into a single row.
A machine learning system fed with this data predicts outputs from inputs, such as the chance of a refund. If refunds are high for restaurant XYZ, the business promotes XYZ less.
Three things hold for the lake:
- A cron job pulls the data from the databases periodically, every 30 minutes.
- The lake is immutable: entries are added, existing records are not updated.
- The lake is cheap, on a file storage system such as HDFS.
The Map Reduce Architecture
MapReduce spreads one job over layers of boxes that read from a data lake and produce a data set.
The work is partitioned across different systems, and the hardware that runs it does not matter. If one box crashes, only the data relevant to it is affected: the data is sent again when the box recovers. There is no single point of failure.
The task is broken into pieces:
- The filter layer. All of its boxes filter.
- More layers, which map or expand.
- A data set. Everything is collected into a single database, whose schema is defined by the people who want the data.
The data set goes to the downstream systems, such as the machine learning system.
Multiple processes run on the same data lake and the same hardware and give different outputs. One filter and reduction produce data set one for the operations team. Another produce data set two for the alerts and monitoring team.
The process is also known as ETL: extract from the lake, transform in the boxes, load onto the downstream services.
Fault tolerance and engineering optimizations
A MapReduce job is fault tolerant because its boxes are stateless and its stages are decoupled.
Whether a stage pulls its work or has it pushed is the wrong question. A data set sits between two stages: an intermediate table that stores intermediate results. Each stage reads from the table before it and writes to the table after it.
The boxes hold no state. There is an input and there is an output, and what a box does is computation. The current state of the box does not define what the output will be.
When a box crashes:
- Another box goes to the input intermediary table.
- It recomputes.
- The output is the same.
It does not matter which box runs a task.
Real time analytics
Real-time analytics answers what a batch job answers too late, such as an alert when the number of trips goes down dramatically.
The data lake pulls from the original sources periodically, and the jobs on it are cron jobs. Querying the lake always takes time. Instead, the services that own the data send events on an event bus, something like Kafka.
- The analytics engine is interested in the trips topic and the payments topic.
- A subscriber condenses two events into a single event on a separate queue, a topic of its own.
- The analytics engine pulls that data and finds trends.
- The decision goes to an engineer or to an operations executive.
This is a streaming architecture. Its queues are temporary, simple operations are easier in a stream, and some loss of events is okay: availability matters more than consistency here. Complex operations, such as ordering all deliveries by price, are easier in the batch process, which loses nothing.
The combination of streaming and batch processing is the Lambda architecture.
Ranking restaurants in real time and batch
A batch job answers a question that can wait: the number of deliveries that failed because the delivery time was too long.
The payment service says the status is customer cancelled. The delivery service says the predicted delivery time was greater than X. The two databases go through a MapReduce job:
- A join of the two.
- A filter on the delivery time being greater than X.
- A filter on the status being cancelled.
- A reduce, which takes a count of the deliveries.
The job takes up to one hour.
The data set goes to the machine learning engine, which finds correlations: a delivery executive with less than two months in the company, or a gourmet restaurant that takes long to prepare. Correlation is not causation, so human beings look at the result.
Sources of data split in two. Real-time data goes as events to the downstream systems, which react immediately. Data persisted in the data lake goes through the long MapReduce process, and its final result is used by the downstream systems to generate insights.