System Design & Distributed Systems Blog
Read by engineers from
Vector Databases vs. Relational Databases: Key Differences and Use Cases
What are the key differences between vector databases and relational databases? This blog looks at their unique features, use cases, and limitations to help you choose the right database.
Scaling Memcached to 1,000,000,000 QPS - Replication vs. Data Sharding
When scaling out caches or database servers, we need to decide how to spread out our data. Should we spread it into separate shards, or club them together in replicas? In this blog, we learn the diffe...
System Design of a ticket booking system
How does a ticket booking system manage thousands of concurrent requests to different events in real time? This blog explains how booking systems ensure exactly-once consistency guarantees with differ...
Pass or Fail: How are technical interviews scored?
This is how interviewers score software developers during interviews, on coding, problem-solving and communication.
Google's PlanetScale Tracing System - Dapper
Dapper is Googles engineering solution to the problem of - How do we trace petabytes of request data generated every day. This blog dives into what Dapper offers, and why it is an engineering marvel.
System Design Components Cheatsheet
This is a table of system design components and their corresponding example technologies. This table comes in handy during interviews and team discussions.
Meta's time series compression algorithm reduces memory requirements by 12x
The engineering team at Meta used a practical difference-of-differences algorithm, to reduce their time series memory footprint from 16 bytes to less than 2 bytes.
Concurrency Patterns for Senior Engineers: Part IV
This blog is a great read for senior software engineers looking to use concurrency design patterns to their advantage. It describes a variation of write-ahead logging.
Concurrency Patterns for Senior Engineers: Part III
Dive into an advanced concurrency pattern called Request Hedging, thats used by Amazon DynamoDB and Google Zanzibar to limit worst-case request latency and avoid thundering herds.
Timeseries Databases: How business metrics are analyzed for anomalies and optimizations
Businesses adapt to changing conditions. And the faster they know about these conditions, the better they are at adapting to them. Sales and operations teams rely on accurate reporting to make such de...
System Design Case Study: How WhatsApp scaled to 100 million daily international calls
International calls are notoriously slow. After the pandemic, Meta had to scale its international calling capabilities to over 100 million daily. What algorithms and infrastructure powers this scale? ...
Why Twitter's write-heavy systems pose new challenges
Explore how Twitter's write-heavy systems present unique challenges, as detailed in a 2020 Carnegie Mellon research paper. The blog post highlights the surprising impact of cache object quantity and s...
AWS Zelkova: "Secure" systems are difficult to build
Understand how AWS Zelkova helps secure systems by analyzing and managing millions of request configurations daily. The blog post explains how it identifies unsafe configurations and uses caching and ...
India's contribution to Open Systems: The Beckn Protocol
Understand the challenges of data ownership, privacy, and portability in social media platforms. The blog post introduces the Beckn Protocol, India's contribution to improving these issues in open sys...
How Amazon handles Massive Scale in real-time through Cyber Mondays
Learn how Amazon powers super-fast recommendations using user-item interaction matrices and similarity calculations. The blog post explains how caching and horizontal scaling are used to improve perfo...
Database Blogs that will blow your mind
Explore a curated list of 21 excellent blogs on database engineering, categorized by specific topics. The blog post includes resources on database replication, in-memory databases like Redis, and NoSQ...
Diving into a new cache eviction algorithm called SIEVE
Discover a new cache eviction algorithm called SIEVE, which adds elements at the head. This blog post explains how this approach can improve cache performance and efficiency
How LinkedIn computes profile distances
Explore LinkedIn's approach to computing profile distances using bi-directional BFS and concurrency design patterns. The blog discusses the challenges of scaling, data caching, and latency reduction i...
How to Read Engineering Research Papers
Learn effective strategies for reading and understanding dense engineering research papers. The blog post offers a step-by-step guide to extracting key insights, taking notes, and retaining knowledge ...
How Uber scaled its engineering team and microservices
Learn about Uber's journey from a small startup to a global giant, focusing on how it scaled its engineering team and microservices. The blog post details the evolution from a simple MVP to a complex ...
Most popular AWS services [for a software engineer]
Discover the top 20 most popular AWS services for software engineers, from Neptune Graph Database to Kinesis Streams. The blog post provides a brief overview of each service and its primary use case
Security in Distributed Systems: The benefits of OAuth
Explore how websites are simplifying user authentication by outsourcing it to larger platforms like Google. The blog post discusses the benefits of this approach, including improved user experience an...
This load balancing algorithm powers Netflix, Google and Uber systems
We describe how Netflix reduced its connection count by a factor of 10 using the right load balancing algorithm. The post describes different load balancing algorithms, and how each solution iterative...
System Design of DigiYatra: Ticket Verification for 1 billion users
Digi Yatra is a biometric boarding system using Facial Recognition to provide a seamless check-in experience at airports. Ticket verification are reduced with existing infrastructure at Airports. This...
System Design of IRCTC: Ticket Booking at Planet Scale
The Indian Railway Catering and Tourism Corporation serves over 8.5 billion passengers annually. Find out how it manages a network of 13500 passenger trains and 7300 railway stations.
Advanced Debugging Tools by Meta Engineering: FBAR and HawkEye
Meta runs on over 100 million lines of code. Finding the root cause of an error is like searching for needles in a haystack. We see how their engineering systems evolved over 10 years to solve this ha...
Matching 25 million trips per day: The System Design of Uber
Uber matches millions of riders with drivers efficiently, affordably, and quickly. It is a mix of product considerations, technical requirements, pricing, and graph algorithms. This blog shines some l...
Rising to TechLead: How software engineers carve a career in tech
Explore the strategic steps: impactful projects, leveraging strengths, fostering team success etc... that will help you rise to the position of tech lead. We explain how to balance technical prowess w...
Strong Consistency in Distributed Systems: The Paxos Algorithm
Explore the complexities and importance of the Paxos algorithm in achieving strong consistency in distributed systems. Learn how Paxos addresses challenges and find out how it is implemented in real-w...
The maximum TCP socket connections possible on a single server
The blog challenges the misconception that the number of socket connections a server can handle is limited by the number of ports, explaining that sockets encompass more than just ports. It clarifies ...
How Git Scales to 500 Million Repositories: Algorithms under the hood
Discover the inner workings of Git, the most popular version control system for software engineers. We understand the algorithms that power its data storage, directory structuring, and version history...
The lost art of database optimisation with Stored Procedures
In an era when cache deployment was a challenge and RAM came at a premium, developers ingeniously streamlined app performance with stored procedures. By executing operations within the database, these...
Database Indexing Strategies: How NoSQL outperforms traditional indexing
Discover why NoSQL databases often offer faster performance than SQL databases due to their approach to indexing. While SQL indexes utilize binary search on individual fields, NoSQL databases leverage...
Meta makes Python 2% slower with Immutable Objects
Meta has recently introduced a groundbreaking concept to Python known as immortal objects. These objects include a special flag that prevents them from being deleted by the garbage collector. By exemp...
Scaling Graph Algorithms: System Design of Google Maps Routing
Explore the intricate system design behind Google Maps' industry-leading routing engine in this insightful article. From graph algorithms to real-time data utilization, uncover the core principles and...
Eventual Consistency in Distributed Systems: Types, Tradeoffs, and Engineering
Explore various eventual consistency types and their tradeoffs in distributed systems. See how software engineers leverage these concepts to build resilient, scalable architectures.
Inside the mind of a Senior System Engineer: What differentiates veterans from newbies?
This blog is for mid level and senior engineers looking to add value to their teams technical discussions and design reviews. We look into the best practices when designing distributed software system...
Where do you see yourself in Five years? Breaking down Behavioral Event Interviews
Unlock success in interviews with our comprehensive guide to the Behavioral Event Interview (BEI), a powerful technique that decodes past behaviors to predict future performance. Master structured que...
.jpg&w=96&q=75){kind=small}
The best and worst way to learn algorithms: 5 Core Subjects in Data Structures and Algorithms
This data-driven blog tells you the best and worst topics to help you master algorithms as a software developer and competitive programmer with the minimum time investment.
Optimizing Your Tech Hiring Process: 7 Essential Rules for Hiring Managers, HR Professionals, and Startup Founders
This blog is for hiring managers, HR professionals, and startup founders looking to recruit the best tech talent at the affordable rates within constrained timelines.
Building Google Calendar Recurrences with the Prototype pattern
The Prototype Design Pattern: Simplifying the low level design of Google Calendar
Concurrency Patterns for Senior Engineers: Part II
This blog is a great read for senior software engineers looking to use concurrency design patterns to their advantage. We examine list partitioning, using the case study of a notification system.
Concurrency Patterns for Senior Engineers: Part I
This blog is a great read for senior software engineers looking to use concurrency design patterns to their advantage.
Say Goodbye to Memory Overloads: The Low Level Design of On-The-Fly Recurring Events in Google Calendar
This blog post delves into a fascinating design pattern of creating recurring events in Google Calendar. The brute-force method of creating multiple instances of the same event is infeasible. The post...
Leading Software Teams- Five essential areas that senior software engineers must master
A guide for experienced software engineers who want to enhance their skills and career prospects. It covers various areas of focus such as task management tools, code management tools, API design, cle...
From Prep to Offer- 9 Tips for Software Developers to Nail System Design Interviews
Tips to ace the system design interview, such as understanding the problem, communicating your thought process, breaking down the problem, and not getting bogged down in details. Candidates with prior...
White papers that are worth your time
Some awesome white papers from Google, Amazon, Meta and Apache. These white papers are worth your time if you are a software enthusiast or a tech geek.
Eliminating Head of Line Blocking - The Evolution of HTTP Protocols
Head of line blocking has been a persistent issue with HTTP protocols, causing delays and frustration for users. In this blog post, we explore how HTTP 2.0 and the latest version, HTTP 3.0 (also known...
10 proven ways of convincing your engineering manager
Validating engineering ideas is crucial before implementation. Learn how to minimize risks and biases and convince your manager with data-backed arguments by following tips from product managers.
Lessons learned from a systems failure - The importance of dynamic rate limiting
Learn about dynamic rate limiting and why it's crucial to ensure your systems work and thrive under fire, especially during peak periods. Read on for more insights and valuable lessons from a systems ...
Amazon Behavioral Interviews - What to Expect and How to Prepare (with a dash of fun)
What are Amazon behavioral questions, and why are they important? Learn how to prepare for them and ace your next interview.
Google Behavioral Questions - What They Are and Why They Matter
What are Google behavioral questions, and why are they important? Learn how to prepare for them and ace your next interview.
UPI Payments System Design Overview
How does UPI work as a real time highly scalable software system, and what are some underlying design principles we can learn from?
How should a system handle failure?
It is clear that Black Friday the busiest shopping and traffic day of the year. Given its importance to traders, how do we handle a system crash?
13 Myths About SYSTEM DESIGN Exposed
13 myths and misconceptions that system designers must face and overcome.
Low-Level System Design of an Inventory Management System like Amazon/Myntra/Flipkart
Let's say you order a product at Amazon during a festive sale(Amazon Great Indian Festival Sale ). How are these products stored and how do they use inventory by automating end-to-end production, ...
Let's Get started with System Design
The interesting fact is that there is no fixed solution when you design a system. Each solution to designing a system has more than one way. So, how do we approach a question related to designing a sy...
Five less known important system design questions
A good developer knows how to solve well known problems. A great developer knows how to stand out. These lesser known system design problems will test you in a new way.
Cache - Fundamentals
What are the different types of caches? How do we pick the best write policies for our use cases? And what replacement policies does Memcache use to improve performance?
Frequently Asked System Design Questions in MAANG Interview
This article contains all the frequently asked system design questions and also their resources.
Workflow Management platform like Airflow and Flink
Workflow management that processes data in real-time and in batches. Engineering optimizations needed to make the system fault tolerant.
Top System Design Questions asked in Big Tech Companies
These are popular curated system design questions, hand-picked to help you solve a diverse set of engineering problems at work and in interviews.
Single Point of Failure
What is a Single Point Of Failure? How do we identify it in a system and how can we avoid it?
Horizontal v/s Vertical Scaling
How do we scale systems to handle a large number of requests? Different methods of scaling and their advantages and disadvantages.
Consistency in Distributed System
What are the different types of consistency levels? What is transaction isolation and how do we implement different types of transaction isolation?
Event Driven Architectures
What is an event driven architecture? How is it different from request-repsonse architecture? And what are its advantages and disadvantges?
Cache - Low-Level Design
Designing a cache with asynchronous processing, read your own write policy, request collapsing and many other features
Data Replication in Distributed System
How does data replication ensure availability, what is master-slave architecture, and solving the Split-Brain Problem.
Designing a Live Video Streaming System Like ESPN
How would you design a fault-tolerant video streaming system like ESPN? And what are the protocols and services that make it possible?
How to design good APIs?
We will learn what is an API, What is a good API and how to design them?
Computer Networks Fundamentals
The different layers of computer networks, TCP, UDP. Communication Standards like REST, and GraphQL and versions of HTTP and the problems solved by them.
What is database sharding?
How would you make your relational databases scalable and available and what are the trade-offs that you should keep in mind?
What are NoSQL databases ?
What makes NoSQL databases different and what are the tradeoffs compared to relational databases.
How Netflix onboards new content onto their platform
Video onboarding, processing and storing at Netflix is complex. We pick each part of this process and dive in detail.
Distributed Rate Limiting
Rate limiting with multiple microservices is tricky, and can lead to cascading failures if done poorly. How is a global rate limiter implemented at scale?
High Level Design for Instagram News Feed
How would you design the backend for Instagram Feed? Persisting posts, comments, likes, authentication, images and notifying users.
Migrating from a Monolith to Microservices
Moving from a monolith architecture to microservices is complex, but sometimes worth it. Here is a detailed plan on how to do so.
High-Level Design for WhatsApp
Internet calling, VOIP and PSTN. Bill users per minute, regulations of different regions, and pick the most efficient provider are discussed here.
Low Level Design for Event Bus
What is an event bus, how do we handle concurrent events, ordering, and idempotency?
WhatsApp Calling - High Level Design
How can you make a phone call over the internet? What is VOIP and PSTN? How do you bill users per minute, following regulations of different regions, and pick the most efficient provider for each call...
Low Level Design of Splitwise
Simplified Transitive Payments, graph algorithms and group payments with code.
High Level Design for Chess Website
How would you design a large scale distributed chess website? Matching users, minimising latency, stopping cheating and game management are explained here.
How can we design an audio search engine?
Given an audio signature, how can you match it to a list of sounds in your database?
Data Consistency and Availability in distributed systems
This article discusses consistency in distributed systems and the challenges that come with it. It covers the Single Point of Failure, Scaling, and Latency issues with a single server and explains how...
High Level Design for Gmail
How would you design an emailing service like Gmail? Registering users, authenticating them otps (Two-factor auth), service deployments and group mailing.
High Level Design for Tinder
Design a large scale distributed system like Tinder. Location based recommendations, profile images, direct messaging and more explained here
What is a load balancer?
A fundamental component of large scale distributed systems, here is how a load balancer sends requests from clients to servers efficiently.
Problem solving in the real world
How are problems solved in the murky world of software development? Is it like in the interviews and coding contests, or is it a little luck with skill?
Data structure and algo usage in software development
Ever wondered why you are solving hundreds of problems on Leetcode or being asked to start competitive programming? This blog goes down the rabbit hole.
Advice before your first day as a software engineer
First job as a software developer? Looking to make a mark, but don't know where should to start? Here are some pointers.
Five things you should know as a software engineer
Stand out as a good developer. Competence builds trust in managers, and young engineers value them. To build this reputation, here are five tips.
Prepare to answer any design questions in a technical interview
It takes a few months of dedicated effort to strengthen one's system design skills. Here are some things to keep in mind.
