lesson 6 of 9 · how a model works

Picking the next token.

The last step scores every token the model knows. Softmax turns the scores into probabilities, and temperature decides how boldly one is picked.

new words

Logit: The raw score the LM head gives one token. Any number; higher fits better.
Softmax: Turns scores into probabilities that add up to 1: e to each score, divided by the sum.
Temperature: A number every score is divided by before softmax. Low makes the top token surer; high spreads the chances.
Top-p: Pick only among the likeliest tokens whose probabilities add up to p, such as 0.9.

the lesson

After the last layer, the LM head takes the vector of the last token and scores it against every token of the vocabulary: 128,256 scores in Llama 3. An example with a vocabulary of only four tokens, after Hi, how are you:

Token	Logit	e to it	Probability
?	3.0	20.09	0.44
doing	2.6	13.46	0.30
today	1.9	6.69	0.15
feeling	1.6	4.95	0.11
Sum		45.19	1.00

Temperature divides every logit before softmax. At 0.5 the logits double, and ? rises to 0.62. At 2 they halve, and the four move closer: 0.35, 0.28, 0.20, 0.17. At 0 the top token is always taken, so the same text gives the same answer.

All the chance in an answer comes from this one step. The layers compute the same scores for the same text; the pick is where two answers part. A higher temperature takes a less likely token more often, which reads as more creative. A lower one reads as more careful.

Top-p cuts the list before the pick: it keeps the likeliest tokens until their probabilities reach p, and picks among those alone. At p = 0.7 the example keeps ? and doing, 0.74 together, and today and feeling can no longer be picked.

The four probabilities of the example, drawn to scale. A real model has a bar for each of its 128,256 tokens, most of them near 0.

try it

Move the temperature. Your browser computes the four probabilities again from the same four logits.

Temperature: 1.0

Probability	Token	Logit ÷ T	e to it
0.44	?	3.00	20.09
0.30	doing	2.60	13.46
0.15	today	1.90	6.69
0.11	feeling	1.60	4.95

T = 1.0: each logit ÷ 1.0
sum = 20.09 + 13.46 + 6.69 + 4.95 = 45.19
? = 20.09 / 45.19 = 0.444

you should see

At 1.0, the table of the lesson: ? at 0.44.
At 0.2, ? passes 0.85; at 0, it is 1.00 and the other three are 0.
At 2.0, the four lie between 0.17 and 0.35.