This chapter explains two key optimizations that improve attention computation in large language models.
Attention needs a KV Cache. As sequences get longer, this creates memory bottlenecks and slows down inference.
Paged Attention handles this by splitting memory into chunks or "pages," loading only the necessary ones during computation.
Together, these techniques allow models to handle long inputs and generate responses faster while using less memory.