Vector compression

What this lesson covers

This lesson looks at how vectors are stored so that the documents relevant to a query can be found quickly. The notes frame it as part of retrieval augmented generation, where documents close to a query ground the response. Relevance is measured as the Euclidean or Manhattan distance between the query vector and a document vector, and a document counts as relevant when that distance is below a threshold T. Comparing the query with every document is order N, and limiting each coordinate to the threshold barely shrinks the search space once there are many dimensions. The video introduces two techniques. The first is compression: if each dimension takes two bytes, a vector of n dimensions needs 2n bytes, but mapping each value into four ranges (0 to 25, 26 to 50, 51 to 75 and 76 to 100) needs only two bits per dimension, which the video calls nearly 90% memory savings. The second is clustering: vectors are grouped into clusters, each represented by a centroid, so a query is compared with the centroids first and then with the documents in the chosen cluster. In vector database terms this is quantisation, and the two can be combined by clustering compressed vectors. The cost is accuracy: compression loses information, and the most relevant document may sit in a cluster that is never searched, a trade the video says is worthwhile as systems scale.

• A document is relevant when the Euclidean or Manhattan distance between its vector and the query vector is below a threshold T
• Comparing the query with every document vector is order N, which does not scale
• Mapping each dimension from two bytes to two bits over four value ranges saves nearly 90% of memory
• Clustering represents each clump of vectors by a centroid, so a query is matched to a cluster before its documents are compared
• Quantisation trades some accuracy for speed: compressed vectors lose information and the best document may sit in another cluster