#4: Debugging in the trenches

What this lesson covers

This lesson starts with a support problem: the founder reports many failed payments. The metrics show the payment gateway failing at its expected rate of about 2 percent, so the real issue is that the team cannot find the root cause of individual failures quickly. The first fix is logging. Each incoming request, each failure-prone operation such as a database write or an external call, and each response is logged with a request ID, so a customer's email can be traced to an order ID and then through the logs. Amazon CloudWatch is suggested because it centralises logs and supports regex queries. The second fix is observability: dashboards of sales, visits and registrations that let the founder spot anomalies, such as daily sales dropping from around 100 to 10, starting with a free tool like Google Analytics before moving to Power BI, Tableau or custom dashboards. The lesson then turns to prevention, examining an architecture of a CDN, S3, a single monolith server and integrations with Shopify, PayPal and Stripe for single points of failure. It covers backup databases, the built-in reliability of managed serverless compute and CloudFront, and backup payment providers. Where a dependency like Shopify has no practical backup, it recommends graceful degradation with a clear, readable error message for customers and support teams.

• Log every request, risky operation and response with a request ID to trace issues end to end
• Centralised log stores such as CloudWatch allow searching logs with regular expressions
• Dashboards of sales and visits help spot anomalies quickly
• External dependencies on the critical path are potential single points of failure
• When a backup is impractical, degrade gracefully with clear error messages