This chapter covers how speculative decoding helps reduce latency in language model responses.
Early versions of ChatGPT were slow, with responses appearing word by word.
Most words in a sentence are predictable and easy for the model to guess.
Only a few tokens actually require significant computation.
Speculative decoding lets a smaller, faster model generate several candidate tokens quickly.
These tokens are then verified or corrected by a larger, more accurate model.
This parallel process reduces response time without sacrificing output quality.
The method improves user experience by making responses feel faster while keeping the model efficient.