Persistence in a time series database
Monarch and Gorilla keep their time series in memory to be fast. To recover after a crash, both persist changes as logs.
Each fix creates the next problem:
- An in-memory node that crashes loses its time series. The changes are persisted.
- A database suits a store that is queried. This store is only used to reconstruct the in-memory database, so logs are enough.
- The logs live in a distributed file store, replicated across zones. Monarch uses Colossus, the next generation of the Google file system. Gorilla uses GlusterFS with a replication factor of 3, and its paper says HDFS would have been fine.
- Logging every event flushes too many metrics to disk. Gorilla holds about 64 kilobytes, roughly one second of data, and then pushes it to the log. A crash in between loses that data.
The logs are not perfectly consistent, by choice. Gorilla trades consistency for high throughput. Monarch trades it for availability: if Colossus does not respond, Monarch continues, because Monarch is what reports that Colossus is failing.
Tradeoffs: choosing a time window
Monarch and Gorilla diverge on one decision: how much data stays in memory.
Facebook says its most important time series queries fall in the last 26 hours. Gorilla keeps only 26 hours of data in memory. Everything older goes to a persistent time series database, queried through Hive. That store has high latency and very low cost.
Monarch makes no such mention. It stores aggregations, on the reasoning that a fault is usually visible even after data is rolled into a single bucket. The aggregated data is highly compressed and still costs petabytes of memory. Gorilla stays in terabytes.
The two assumptions buy different things:
- Gorilla answers questions about recent data quickly and cannot run a very long time series analysis in memory.
- Monarch gives metrics and predictive analysis over a much larger series, such as yearly trends, and holds up better against future requirements.
The cost of Monarch is about 1000 times the memory, and a much more complex architecture than Gorilla's.
Load balancing in Gorilla
Gorilla balances load through an external system called a shard manager, which maps ranges of keys to hosts.
The key space is cut into ranges: 0 to 500, 500 to 1000, and so on. Range R1 is handled by host 1. A request for key 29 asks the shard manager where to go.
Scaling out follows a chain:
- A new host, H5, enters and the shard manager assigns it a range, R5. Existing ranges shrink.
- A host that gives away part of its range drains out: data and load leave it. A host that takes a range drains in.
- H5 cannot serve at once, because its in-memory database is empty. The other hosts drain out only after H5 has drained in, which takes about 5 minutes.
A crash is handled the same way. If H2 is lost, the shard manager reassigns R2 to the remaining nodes. They drain in by reading the logs stored in GlusterFS, and then serve requests.
High availability needs fault tolerance: an answer for the range whose host has crashed.
Replication and fault tolerance in Gorilla
Gorilla stays available through a host crash by keeping two copies of everything, at two levels.
- Two hosts serve the same range. A cluster sits in one data center and holds two independent instances of each host. Every write goes to both, so the logs are duplicated too. No attempt is made to keep the two logs consistent, and the data may differ between them.
- Several data centers. If an entire cluster goes down, read queries are redirected to a different data center. The data is still served. It takes a little longer, and the data center that takes the load may run hot.
These two characteristics make the system highly available.
For a very consistent view there is a time series database that stores all the writes, queried through a Hive layer. Gorilla is for fast queries and anomaly detection in real time, with very low latency.