Observability in Distributed Systems

Introduction to Observability

Observability means an engineer or an operations person can look at a system and tell whether it is functioning as expected or something is wrong with it.

Several topics come under it: logging, monitoring, anomaly detection and alerts, root cause analysis and incident reporting. Anomaly detection and alerts are close to each other.

A system that is monitored effectively earns more trust:

1. Errors are found quicker.
2. Business problems are notified to the right people quicker.
3. Some errors are resolved automatically.

A robust and fault tolerant system needs excellent observability. The chapter covers each topic in detail, so that an engineer can name the parts that are missing in an organization and the parts that can be improved.

Logging - Best Practices

Logging is the simplest form of getting data from a system: a log line is written to a file whenever a request comes in or the system does something.

A log line is like a println statement in Java or cout in C++. A typical line holds:

1. The timestamp of the event.
2. The level: I for information, W for warning, E for error. A system-breaking error at level one may create an incident report.
3. The file and line that wrote it, such as content.go 264.
4. The log line itself, such as user content progress created, with the ID of the content.

Personal user details are a privacy decision. In financial systems, where engineers see transactions, the identity is hidden: only a user ID is printed and the email is protected, or the user ID is not printed at all. Passwords are blurred out.

For debugging, the request ID is printed in every log line, and a grep for it returns the series of log lines related to that request.

Monitoring Metrics

Monitoring takes metrics, meaning numbers, from a system and tracks them across time.

The numbers can be about the machines: the amount of memory used, the number of I/O operations. They can be about the business: the number of sales, the number of visitors on a page.

Tracked across time, each number becomes a graph, a time series, that a human being can watch. An operations team looks at the numbers daily or hourly. Monitoring is usually manual, and it can be automated.

The example is AWS CloudWatch tracking an elastic load balancer. If the number of requests spikes up, more load balancers are added. If it drops, a load balancer is removed to save cost. The decision is made by a person reading the dashboard.

A data store powers the dashboard. It can be a time series store or plain Postgres. A small organization usually uses the data store it already has. A cloud provider such as AWS sells this ready built, with alerts and monitoring.

Anomaly Detection: Holt-Winters Algorithm

Anomaly detection finds a metric that has left its normal range, automatically, so that the response to a problem is quick. A business that makes 3,000 sales a day and drops to 200 wants to know the same day.

The example is video retention: the percentage of users still watching at each second of a video. It declines smoothly, with a few spikes.

1. Confidence intervals. Each point carries a range the next point is expected to fall in, and the range narrows as more of the graph is seen. A point outside it is an anomaly. This raises false alarms on known events such as a Black Friday sale.
2. Differentiation. The rate of drop off is almost constant, so its derivative, the acceleration, is constant, and the next derivative, the jerk, is near zero. A large jerk is flagged.

The Holt-Winters algorithm detects spikes even in seasonal metrics. Seasonal changes give small jerks and are ignored. A big spike is caught.

Root Cause Analysis

Root cause analysis finds what went wrong in a system when there was an anomaly or an incident: from the symptom, the cause.

There are three ways to do it.

1. Manual. A person looks into the log lines and the metrics and builds an incident report: when the problem occurred, what the root cause was, and the technique used to find it. One technique is the five whys. Memory consumption was high. Why? Many PDF files were generated together. Why? A cron job generated them hourly. Each answer goes a level deeper. With several contributing factors the line of whys becomes a tree.
2. Automated. It is hard to do, error prone and fast. When sales and visitors both go down, sales depends on visitors, so the independent metric is the one to fix. Principal component analysis and the Spearman coefficient measure how correlated two metrics are.
3. Outsourced. Amazon CloudWatch and Amazon SageMaker detect anomalies. One of SageMaker's algorithms is isolation trees: sparsely populated branches of a population are anomalies.

Conclusion to Observability

Observability in a distributed system grows with the organization that runs it. The chapter covered three parts: logging and monitoring, anomaly detection, and root cause analysis.

1. Small and medium organizations collect data. The focus is on collecting the data and interpreting it manually.
2. Medium to large organizations detect anomalies automatically, and may try to resolve some of them.
3. Large organizations try to resolve quite a few anomalies and add root cause analysis, so that their teams function effectively.

The problem of error handling is scaled out this way.

Anomaly Detection

Anomaly detection on the server side manages the health of a system: the servers publish their metrics to an engine, and the engine flags an anomaly and alerts an engineer.

False positives are allowed far more than false negatives. An engineer investigating a false alert is much cheaper than a real alert being missed.

Each service has its own metrics, so a service mesh with a sidecar gives uniformity. The sidecar is a library for common functionality, and it publishes the metrics of every application to the engine.

Finding anomalies in the time series data:

1. Fixed limits, from historical data. They ignore the trend: a metric that has gone down for months hits the low pass filter.
2. Dynamic low pass and high pass filters that stick close to the graph. A sharp change is flagged. They ignore time: the spike on the 25th of December is normal.
3. An isolation tree, which is a decision tree. It partitions the data by features, and a point set apart in very few partitions is an anomaly.