Distributed data consistency
Consistency means that multiple copies of the same data match each other. It is the C in the CAP theorem and the C in database transactions.
The example is Facebook in its initial days: a single server at Harvard with a single copy of data. User A creates a profile, user B asks for it, and the server returns it from its database. The answer is always right.
A single server has three problems:
- It is a single point of failure. A network outage or a power loss takes the system down.
- Vertical scaling costs a lot and hits a limit.
- Latency. A student at Oxford connects across a continent, about 100 milliseconds away.
A second server at Oxford, with no communication between the two, serves each campus its own data. No data is shared.
Sending a Harvard student to the Oxford server for Oxford data shares it, and keeps one copy. Vertical scaling is no longer a concern. The Oxford server is still a single point of failure, and the latency between continents is still high.
Tradeoffs - 2 phase commit protocol
Two phase commit is a protocol in which a leader asks its followers to prepare a transaction and then to commit it.
- The leader gets an update and sends a prepare request to its followers.
- Each follower begins a transaction, writes the statements, keeps the row locked and sends an acknowledgement. Only the commit is remaining.
- The leader commits to its own database and sends a commit message to everyone.
- Each follower commits and acknowledges.
If the leader does not get all the acknowledgements for the prepare, it fails the transaction and tells the followers to roll back.
A follower cannot roll back by itself on a timeout. If a commit message fails and the follower rolls back, a get returns C on the follower and C++ on the leader. So the leader keeps retrying the commit until it is acknowledged, and the transaction ID makes the retry idempotent.
Until then, read and write operations on the row are blocked. The system is totally consistent, and the cost is that it is not available.