Dremel is the system inside Google BigQuery that lets product managers and CXOs query petabytes of data interactively.
Earlier, a query ran as MapReduce on the Google File System and took minutes, hours or days. A product manager asked an engineer to write the job, waited two or three hours for the answer, and waited one more hour to change a filter.
Dremel was built to give five things:
- Interactive analytics. A filter is changed and the answer comes in tens of seconds.
- Short response times. Trillions of records are scanned in tens of seconds.
- SQL queries, in 2010, when NoSQL was at its height. Product managers and data analysts are comfortable with SQL.
- Columnar storage. A table is stored column after column, so a search on the city reads the city column alone.
- Fast aggregations, a sum or a count, with near linear scalability. More nodes take a higher load and latency stays flat.
Borg inspired Kubernetes, the Google File System inspired Hadoop, and Dremel inspired Apache Impala and Apache Parquet.