What is a time series database?
A time series database stores data useful for tracking metrics.
Two papers describe such a database: Google Monarch, from 2020, and Facebook Gorilla, from 2015. Each made its own decisions and had its own outcomes.
A time series database is used for five things:
- Metrics tracking.
- Anomaly detection: whether something is out of range or within range.
- Predictive analytics, such as how many machines are needed.
- Dashboards, because some analytics are too complex to predict.
- Debugging or tracing, a part of observability. Monarch shows that things are going wrong and then the pattern of a particular request.
The design of the database is one subject and the features built on top of it are another. Metrics tracking depends on how data points are stored. Both papers mention anomaly detection. Predictive analytics, dashboards and tracing can live in a separate system.
Requirements of a time series database
Facebook and Google built their own time series databases instead of using OpenTSDB, InfluxDB or Prometheus. Scale is the reason that stands out, and the two share three requirements.
- Scale. Gorilla's 2015 paper claims tens of millions of data points per second. Run for a day, that goes into terabytes, and the scale had doubled by the time the paper was published. Google stores petabytes. The known solutions are databases on disk, which do not hold up here.
- Low latency. Metrics are tracked, anomalies detected and requests traced in real time, so queries have to return quickly. Databases on disk do not work out.
- High availability. A time series database tracks anomalies, and any system it depends on can pass its own anomalies into it. If Monarch depends on Colossus and Colossus goes down, Monarch cannot report the anomaly in Colossus. External dependencies are reduced as far as possible.
Both databases take similar approaches and reach similar conclusions, with a few fundamental differences.
Data schema of a time series database
Both Monarch and Gorilla treat low latency as a very important factor. Three ideas lower latency: a customized data structure, a cache, and operating system hacks of the kind Kafka uses. The lesson takes the data structure.
Both databases store a key with a time series as its value. The key reads like a path in a file system: a service, a metric, and more qualifiers for more detail. A time series is a set of timestamps, each with a value.
The two differ in the timestamp:
- Monarch. A timestamp is a bucket, and the buckets are equidistant. Data points that arrive inside a window are rolled into one aggregated value. A key holds a histogram of values. Some value is lost.
- Gorilla. Every data point keeps its exact time and its exact value. Nothing is rolled up or aggregated.
Monarch compresses better: 1000 points in one second become one value. Gorilla allows a deeper dive into the data.