Primary Replica Architectures
A primary replica architecture sends every write to one database, the primary, and copies each operation to a replica that takes over when the primary goes down.
A server with a single database has a single point of failure. A restart leaves the clients without their data, so a replica, also known as a backup, is kept. Writes go to the primary and are copied to the replica. Reads go to both.
In read-intensive applications the writes are infrequent, 1% or 5% of the SQL queries. When the primary goes down the replica is promoted to primary, and a new replica is added as a copy of it.
Benefits:
- Fault tolerance.
- Improved read speeds, with reads distributed among two or more nodes.
- Simplicity. A cloud solution provider does it.
Drawbacks:
- Consistency. An operation that adds 10 to a balance is in flight: it is in the primary and not yet in the replica, and concurrent reads see different views.
- Slower writes. Each write also goes to the replica. Latency is traded for fault tolerance.
WAL and Change Data Capture
A write-ahead log is the log of operations a primary applies, copied to a replica. Change data capture sends each change as an event that a subscriber transforms.
Both databases start in the same state. A new write, x = x + 10, is applied in the primary and sent over the wire to the replica, which applies the operations one by one. The replica acknowledges operation 4 and asks for number 5.
Because it is a log, the operations are executed sequentially and rolled back, as in transactions. Only operations are sent, which keeps the transfer small.
Two drawbacks:
- A replica that does not catch up is a consistency issue.
- Context. An entry with the timestamp Friday 10 a.m. arrives three minutes later, and the replica has two timestamps to choose from.
Change data capture handles operations with context. It works like a message queue: an event is published, and the subscriber decides what to do with it. It suits different types of databases, such as write-optimized Cassandra copied into an RDS for relational reads.
Write Amplification and Split Brain
Split brain is two primaries that both believe they are correct and hold different copies of the data. Write amplification is one write turning into a write on every replica.
With two primaries, a network partition stops a write on one from being reflected in the other. One server sends x plus 10 to its primary, another sends x minus 10 to the other, and x has two values.
Last writer wins overwrites somebody else's operation, so usually an admin reconciles manually. An odd number of primary nodes prevents it: when the cluster splits, the majority settles on one value.
With many read replicas, x = x + 10 is copied into every copy. Waiting for all of them makes fault tolerance worse and latency very high. Not waiting still means four operations in place of one. That loads the primary's bandwidth and leads to inconsistencies.
The fix is for a majority of nodes to agree on a value, or a peer-to-peer ring of three or five nodes that reaches consensus through Paxos or Raft.