A vector is a 1D array of values for an object, such that similar objects have a low distance and dissimilar objects a high distance.
A document was first represented by term frequency: the number of times each term occurs, as a 1D vector.
Now a neural network gives the vector. It converts anything into a vector, gibberish included.
A word reaches its vector in steps:
- The token. A word is split into tokens. Eating is split into eat and ing.
- The ID. Each token has an ID, such as 36 for cat. A number maps to a vector directly.
- The vector. Each ID is mapped to a 1D array. Its length is fixed for all the tokens, such as 7000 dimensions.
Byte pair encoding finds the split. The text is split into characters, and pairs that occur together often are clubbed into one token. So ing and er become tokens.
The LLM is a black box. Engineers often store these vectors in a DB.