Throughput vs. Latency

Introduction to design tradeoffs

A design trade-off is one non-functional requirement met at the expense of another.

Uploading pictures on Instagram is one functional requirement with several non-functional requirements:

1. Recovery from a failed upload. If the phone loses charge or internet with half the picture uploaded, the rest is uploaded later.
2. Safety, or persistence. A tsunami that takes down a data center in Japan does not lose a creator's work.
3. Quality. The image is understandable when it reaches the receiver.

These are business requirements, driven by what the customer expects.

Safety is at loggerheads with cost. Persisting the image in 300 places keeps it safe from disaster, and the cost and the time taken to upload become very high. Quality is also at loggerheads with cost, because a high definition image stores more data.

Viewing images brings in response time and the ability to cache, and smaller images cache easily. One copy or three copies of the main image are stored, and 50 copies of the popular images are cached around the world.

Throughput vs. Latency

Latency is the time required to get a response from a server, and throughput is how much of the processing power is spent on the application's own tasks.

A server runs at most 20 instructions per second. A profile service spends 10 waiting on the database and two on garbage collection. The 8 that remain give a throughput of 40 percent.

How often the background task runs sets the trade-off:

1. Background tasks kept for the end. The application runs for 97 seconds and the background task for 3. Throughput is 97 percent, and a request arriving in those 3 seconds is stuck, so worst case latency is more than three seconds.
2. Background tasks in small parts. The task runs 90 times and takes 9 seconds in total. Throughput falls to 91%, and worst case latency falls with it.

Thrashing makes both worse. With very small time slices, context switching eats into the application time. A cache with two entries and requests for A, B and C in a cycle misses on every read.

SQL vs. NoSQL databases

SQL and NoSQL databases differ in consistency, architecture, scalability, sharding and transaction support.

1. Consistency. NoSQL usually has more relaxed consistency. SQL usually has high consistency.
2. Architecture. SQL replicates data through a primary replica architecture: writes fall on the primary, reads on either. NoSQL is usually a cluster, where any node responds or a hash routes the request to a node.
3. Failover. NoSQL has leader election or a consensus algorithm, such as quorum. SQL has an assigned primary, and when it fails the replica becomes the primary.
4. Sharding. NoSQL usually comes with inbuilt sharding: requests from 0 to 100 are split into five ranges on different nodes. SQL usually does not.
5. Transactions. SQL has ACID support. NoSQL usually has none.

NoSQL is designed for high scalability, with fast reads and writes. In SQL the reads are fast and the writes are slower, because they are bottlenecked on one node.

MySQL and Postgres suit relational or transaction support. Amazon DynamoDB, MongoDB and Cassandra suit extremely high scale with denormalized key value pairs.