The embedding matrix.
Each token fetches one row from a big table of numbers. The row is the token's vector: where it sits on a map of meaning.
new words
- Matrix
- A table of numbers in rows and columns.
- Embedding matrix
- The model's first table: one row for each token of the vocabulary.
- Dimension
- How many numbers one vector holds. 4,096 in Llama 3 8B.
- Dot product
- Multiply two vectors number by number and add the results: one number for how much the two share.
the lesson
The token IDs of lesson 2 are only addresses. Llama 3 8B's embedding matrix has one row for each of its 128,256 tokens and 4,096 columns. Token number n fetches row n, and that row is the token's vector.
128,256 rows × 4,096 columns = 525,336,576 numbers
Every number in the table is a weight, set by training. Training moves the rows until the model's guesses of the next token come out right. Words used in the same places end up with rows that point the same way. Nobody names the directions: a direction that means "about money" is one that training found, not one a person set.
The dot product measures how much two vectors share. An example with three numbers, each named for what it might mean:
question person verb
you = [ 0.1, 0.9, 0.2 ]
are = [ 0.3, 0.2, 0.9 ]
you · are = 0.1×0.3 + 0.9×0.2 + 0.2×0.9
= 0.03 + 0.18 + 0.18 = 0.39
Divide by the two lengths, √0.86 = 0.927 and √0.94 = 0.970, and you have the cosine of AI basics, lesson 5: 0.39 / 0.899 = 0.43. Inside the model the dot product does a second job: in lesson 4 it decides how much one token takes from another.
try it
Type this and press Enter:
Compute the dot product of [0.1, 0.9, 0.2] and [0.3, 0.2, 0.9], then their cosine. Show each step.you should see
- A dot product of 0.39, from 0.03 + 0.18 + 0.18.
- Lengths of about 0.927 and 0.970, and a cosine of about 0.43.