Black Friday is the busiest day of the year. The traffic on e-commerce websites on this days is roughly thrice that of a normal day. With the added pressure on software systems, failures become Thanos. You know, inevitable.
As software engineers, we aim to build systems that handle failures gracefully.
Winter is Coming - Ned Stark
Fault and failure
There is a difference between a fault and a failure. A fault is a problem confined to space. It may be a request timeout or a corrupt data entry. Failure is a collapse of our system, resulting in an inability to continue services till it is fixed.
Fault : Pouring a dosa with inconsistent thickness on the pan.
Failure : Burning the dosa to a crisp is a failure.
Creating a zero fault-tolerant system is impossible. However, systems can be designed so that the faults do not cascade into failures.
Use Case:
Let's take an e-commerce app. The user flow involves Ordering, Delivery Management, Invoice Generation, and returning an Order.
Failure possibilities
On Black Friday, the concurrent traffic is enormous. And whatever stops the user from ordering the product is a failure. The usual suspects are:
- Pooof…services go down
- The database slows down
- Service APIs slow down and time out
Detective System Designer will now have to resolve these issues, but how?
Failure Handling
What if the Ordering Service is down, but I have only an hour on the Black Friday sale?
This is a nightmare. There are a few ways to avoid it.
- Scale instances when you are reaching peak load.
This can be done manually or automated. You may also use serverless technologies to offload the problem to the cloud solution provider.
Beware of slow or unacceptable SLAs. Read all scheme-related documents carefully before investing.
- Freeze your code before the sale.
Why? Because most bugs are either regressions or appear in new features. A week or two before the big day, stop accepting changes in your systems. This might be extreme, and the freeze period can be tuned according to your requirements.
- Load Test before the sale.
This problem could have been avoided had we known the maximum load the ordering service could take before the big day. Load testing also tells you what problems you face as your system scales and the potential points of failure.
- Trace requests throughout the system with logs.
A request has an ID. Use it.
Which service failed the request? What went wrong? Is the database okay? How can it be fixed now?
Honestly, getting a system back up within one hour is a hit-and-miss situation. The important thing to do is stay calm and think of short-term fixes.
- Downgrade the quality of responses and fail low-priority requests.
You don't need to show users exactly how many others are also looking at the same product. Send an approximation, or gracefully degrade the experience.
🛠 When in doubt, restart the server" - Julius Ceasar.
What if the database goes down?
The database is the ultimate head villain when it comes to a single point of failure. If the database goes down, your service is living on borrowing time. Once your cache expires, the fun is going to begin.
Since the database is a crucial resource, its queries should be precise and done right.
- Avoid JOINs in your queries.
- Cache popular query results
- We could also increase the cache expiry time.
The next step is reading replicas, global caches, materialized views, etc. This will save some resources for DB, and it will continue to serve user traffic smoothly.
What If the inventory service is down?
An inventory management system holds stock of all the products in the system. Ordering, restocking, and returning need an inventory service update.
To get a detailed idea, check out our blog on the Low-Level System Design of an Inventory Management System like Amazon/Myntra/Flipkart.
What if the inventory service is slow? We live in a world of people with no patience. Even noodles are expected to be ready in 2 mins!
Synchronous calls to inventory service from the order service can result in timeouts and error responses to the user. This leads to blocked calls for other users, and repeated retries from the blocked user.
We recommend:
- Scale bottlenecks horizontally.
- Choose the right persistence technologies (database, file system, or no persistence)
- Having timely monitoring when the service gets degraded.
If the payment service is down or slow
Generally, when we do payments, we do it once. And if there is a failure, we keep retrying again and again.
This might overload the systems. Therefore we could limit the retries to 3. We may also use exponential backoff to spread our retries.
This reduces the burden on the clients connected to us, and also the load on our servers.
———————————————————————————————————
To conclude, if a failure happens, let it happen without cascading. Let's fix it one bug at a time!!
There is a lot more to system failures and handling them at scale. This topic has been discussed extensively in our course.
We offer a structured approach with challenging quizzes and detailed PDFs inside our system design course.
"Don't wait. Get InterviewReady bro" - Albert Einstein.
