lesson 2 of 9 · how a model works

Tokens are learned pieces.

A model never sees letters or words. It sees pieces from a fixed list, learned from text before training began.

new words

Vocabulary: The fixed list of pieces a model knows. Llama 3's holds 128,256.
Byte-pair encoding, BPE: The usual way to learn the list: start from single letters, and merge the pair seen most often, again and again.
Merge: One new piece made of two old ones, such as a + y → ay.
Token ID: A piece's number in the vocabulary. The model reads the numbers, not the letters.

the lesson

An example text of seven words: days three times, pay twice, day once, paid once. Start with single letters. Count every pair of pieces that stand side by side, and merge the pair seen most often into one new piece. Then count again.

Merge	New piece	Seen	Next best
1	a + y → ay	6	d + a, 4
2	d + ay → day	4	ay + s, 3
3	day + s → days	3	p + ay, 2
4	p + ay → pay	2	every other pair, 1

The list now holds six letters and four pieces: ay, day, days, pay. Cut with it, days is one token. payday, a word the text never held, is two: pay + day. paid, seen only once, stays four letters.

A real tokenizer runs the same rule over billions of words until its list is full. A common word becomes one piece. A rare word, a name, or a word in a language the text held little of is cut into several, so the same sentence costs more tokens in some languages than in others.

Digits are pieces too. Llama 3's tokenizer splits a number into groups of at most three digits before it looks up pieces, so the model never sees 1,250,000 as one thing. Exact sums go to a tool. Claude's tokenizer is not published; its API counts the tokens of a text for you.

The word payday, never in the example text, cut by the list the four merges built: six letters become two tokens. Merge 3 makes days, which payday does not hold.

try it

In Claude Code, type this and press Enter:

Add 1,250,000 and 3,498,317 with a command, not in your head. Then tell me in one sentence why a model that reads tokens hands exact sums to a tool.

you should see

4,748,317, from a command Claude ran.
A reason such as: the model reads a number as pieces and predicts each digit, so a tool is exact where the model only guesses well.