A clustered index answers a search query with the closest 10 results, and it finds them by brute force inside one cluster.
Netflix knows the last 10 movies a user saw and has to find the top 10 related movies. A food app has the last 10 meals. These are the search queries.
One query is answered in two steps:
- The cluster. The DB is split into clusters, and the query goes into the nearest one, found through its representation.
- The brute force. Every point of the leaf cluster is compared with the query, and the closest 10 are the response.
Ten queries become the input in two ways:
- One vector. The ten vectors are averaged, or one representation is chosen among their clusters. This is approximate.
- Ten parallel queries. They return a hundred results. The duplicates are removed and the top 10 with the lowest distance are kept.
A horror movie and a romcom are not similar. Their average is neither, so running the queries in parallel helps.