Dremel stores all values in columnar format, and a column compresses because its values repeat.
A table of the numbers 1 to 20 is stored column after column: 1, 6, 11, 16, then 2, 7, 12, 17. A city column holds Mumbai, Mumbai, Delhi, Delhi. At Google scale a value is repeated millions of times across trillions of records. Age runs from 12 to 112, a hundred distinct values, so it is stored as an enum.
Compression saves storage. More data is pulled into memory at a time, and it is pulled faster. This is what makes interactive analytics possible.
It has costs:
- A mapping has to be kept. M M D D is stored as 0011.
- The data has to be ordered.
- Errors have to be corrected.
- Compression and decompression both use CPU.
Decompression happens on the root. Columnar storage allows queries to be fired on compressed data, so nothing is decompressed before the queries are fired.