Monarch's architecture
Monarch answers a query through a tree: a root mixer, zone mixers under it, and leaves under them.
The instances that hold time series are called leaves. A range assigner maps ranges of the key space to leaves. Keys with the same prefix are stored close to each other, so the data for one query is likely to sit in one leaf.
Clients do not connect to the range assigner. A query takes this path:
- It goes to the root mixer, which has a global view.
- The root mixer sends it to the zones best placed to answer. The other zones are not touched.
- Each zone mixer picks the leaves in its zone that answer best.
- The leaves evaluate the query and respond. The zone mixer merges their results, and the root mixer merges the results of the zones.
The same range may be stored by several leaves, so the loss of one leaf does not stop a query. The root mixer looks like a single point of failure and a bottleneck.
System features of Monarch
Monarch protects its tree of mixers in two ways and offers a richer query language than Gorilla.
- Quota management. A really bad query could ask the root mixer to send back petabytes of data. Every node keeps an in-memory quota: the root mixer, the zone mixers and the leaves. One query cannot take all the resources.
- Replication. The root mixer is not a single point of failure. There are several root mixers, and several zone mixers that can answer a query.
Gorilla's query language is very simple. Monarch supports a group by. One key is aggregated per hour and another per minute. When the result is asked for by the hour, the bucket is created on the fly, and both time series come back with the same one hour blocks: minute 60, minute 120, minute 180. Two series with different bucketing can then be compared.
The interpolation hides a gap. A value missing at minute five does not show in the first hour.
Query optimization in Monarch
A field hints index tells a mixer whether a node can answer a query, as a Bloom filter does.
The root mixer does not look at every range a zone handles. It holds an index that says whether a zone can or cannot handle a query. Each zone mixer holds one for its leaves.
The index is built from trigrams. A query for the key GKCS is cut into its runs of three characters, with markers for the start and the end. A node that holds every one of those trigrams can potentially answer. Only potentially: a key that repeats three characters also passes.
The two databases compared:
- Gorilla gives extremely fine grained time series, is real time, and has a simple architecture.
- Monarch aggregates, which shrinks the data and makes patterns easier to find. It runs about 120,000 machines.
Storing only 26 hours of data removes the need for root mixers and zone mixers. Both agree on ranges assigned to hosts, an in-memory database backed by a file system, and delta of delta compression.