Dapper is the request tracing system at Google: it records the path one request takes from the client to the server and every data store, and how much time it took.
A tracing system that slows production is not adopted. Dapper has to be low latency and used by all services, so it is added to the generic client that makes every remote procedure call.
- A span. Each remote procedure call is a span with an ID, a parent ID and an array of events.
- A trace. All the spans of one request carry the same trace ID.
- Collection. Clients write log files to local storage. A background process buffers them. Collectors pull them, convert them into spans and store them in Bigtable, which has an index on the trace ID. Most requests reach Bigtable within 15 seconds.
- Sampling. IO calls are the bulk of the latency, so production samples one request in a thousand and twenty four.
- Adaptive sampling. A service with little volume asks for at least 10 samples per second.