Memory vs. Latency

Memory vs. Latency

A system can lower its latency by spending memory. The best example is a cache.

Fetching a profile from the database takes time, say 100 milliseconds. The profiles of the people who are extremely active are kept in a cache in memory on the server. An active user keeps asking for his profile and his follower count. Both are kept in the cache, and each request is answered from it in 10 milliseconds instead of 100.

The cost is memory. The profile is now stored twice, once in the database and once in the cache, and the cache costs money.

The tradeoff runs the other way too. A smaller cache uses less memory, and more of the responses take 100 milliseconds.

The medium approach has three parts:

1. Keep the hot entries in the cache.
2. Use a good replacement algorithm.
3. Use a good write policy.

Latency vs. Accuracy

Latency is the time a system takes to respond to a request. A system can lower its latency by giving up some accuracy.

Three ways:

1. A cache. The system calls the database and the cache together. The cache answers 30 in one millisecond. The database holds 31 and takes 10 milliseconds. The system returns 30.
2. An approximation. A request asks for the likes of a video. Counting them means reading a queue of database requests and then the database. Instead the system uses earlier data: the video had 10 likes at 1K views, and it has 100K views now, so it answers 1000. The assumption is that the ratio of likes stays the same.
3. A sample. Google Maps knows a road's average speed is 20 kilometers per hour. Out of 500 drivers it reads 50, sees two kilometers per hour, and sends an alert that the road is broken.

The answers are fast and can be wrong. A bank transaction or a balance cannot use them. Increasing accuracy increases latency.

Relations Between Tradeoffs

The parameters of a design trade-off relate to each other in pairs, and each relation is an up or a down arrow on either side.

1. More memory, lower latency, because most things can be kept in memory.
2. More memory, more cost.
3. Lower latency, lower throughput. Throughput is how much of the running time is spent by the application, and it relates to latency alone.
4. Greater cost, lower latency. It is a dotted line: more machines scale the system, but there is no guarantee.
5. Lower consistency, lower latency. A system that is okay with inaccurate data gets very low latency.
6. Lower consistency, higher availability. The two are inversely proportional.

Memory has no real relation with consistency, availability or throughput. Those depend on the policies and the algorithms used.

People using a system do not care whether it is a CA, CP or AP system. They care whether it is good enough for the SLAs.

The trade-offs that bubble up are usually latency, consistency, availability and memory. Throughput and cost are rarer.