Live Streaming Requirement Breakdown
A live streaming service transports high definition video from a sports or cultural event to millions of users who are interested in the stream. Hotstar and ESPN are examples.
The constraint is that the video reaches the users within one minute. The challenges are compute, bandwidth and performance.
The requirements:
- Transform the original video. The 8K video is converted into other resolutions, such as 4K, 1080p, 720p and 360p, and into different codecs for different operating systems. It is a 2D matrix of resolutions to codecs.
- Heterogeneous delivery, to the various devices of users at scale.
- Caching, because performance is tied to the bandwidth used.
- A fault tolerance guarantee, because users are paying.
The first two are specific to this system: the data is video, and the users are heterogeneous. Caching, fault tolerance and performance are engineering guarantees that are default for any engineer. The most critical requirements are the first two.
Video Ingestion and Transformation
Video ingestion takes the live video in as a stream, and transformation converts it into the resolutions and formats the clients need.
A live stream cannot wait for the whole video to end, as Netflix does with an upload. It is ingested over RTMP, which is written over TCP: acknowledgements and ordering, so no data loss. TCP is slower than UDP, but poor quality at the source only gets worse on the way to the user.
The original video is backed up in multiple stores. Then:
- The transformation service has a job scheduler, with tasks such as converting 8K footage into 720p video or 360p HLS.
- It publishes the tasks to a message queue, and worker nodes subscribe and pick them up.
- A worker persists its result in distributed file storage, because a full video in a message queue is a very bad idea.
- The worker publishes an event that it is done.
The transformation service publishes tasks at its own pace and the workers pull at theirs.
Transferring processed videos to end users
Processed video travels from the transformation service to servers around the world, then to a content delivery network, whose edge servers send it to the users.
- Last mile. Users connect over HTTP. An iPhone uses HLS (HTTP live streaming) and other operating systems use DASH. Both have adaptive bitrate: if a user on 480p is able to buffer a lot, the browser sets the video quality higher.
- Own servers. A service such as Hotstar keeps servers in India, the US and the UK, because the SLA of a content delivery network may not match live streaming guarantees. Video reaches them over RTMP, with no data loss.
- Routing. The original server redirects a user to a box by location, and the box gives an endpoint to pull the video from.
If the Indian servers go down, requests are redirected to the US server.
The content delivery network takes care of caching the video. The servers cache the endpoint, with the live stream ID and user ID as the key.