instibrain
Sign in
instibrain  /  guides  /  how a model works

What happens inside the model, in nine short lessons.

AI basics says what a model reads. This series opens the model: one pass from your text to the next token, how the numbers inside it were set, and what it takes to run them. Claude's shape is not published, so the counts use Llama 3, an open model whose maker publishes its shape.

Lesson 1: One pass, one token → AI basics comes first

No mathematics beyond multiplying, adding, and raising e to a power. Lesson 6 has a live toy to move.

the lessons · the words under it

Nine lessons, from your text to the next token.

Lesson 1 is the whole pass. Lessons 2 to 6 walk it step by step. Lesson 7 says how the numbers inside were set, lesson 8 what it takes to run them, and lesson 9 names the kinds of model.

  1. 1One pass, one tokenText, tokens, vectors, layers, a score for every token, one picked; then the whole pass runs again.
  2. 2Tokens are learned piecesA tokenizer merges the pair it sees most, again and again; worked by hand on seven words.
  3. 3The embedding matrixEach token fetches its row from a table of 128,256 rows: its vector, a point on a map of meaning.
  4. 4AttentionEach token takes a share of the tokens before it; three tokens worked by hand, every number shown.
  5. 5Layers and weightsAttention, then feed-forward, 32 times; count the 8 billion weights matrix by matrix.
  6. 6Picking the next tokenA score for every token, softmax, and the temperature you move in a live toy.
  7. 7Training, in four stagesRandom numbers nudged after every guess; pretraining, then three stages to the chat model you use.
  8. 8Running a modelEvery token reads every weight once, so memory decides speed; the KV cache and thinking tokens.
  9. 9Kinds of modelsOne design, three last steps: a chat model writes, an embedding model finds, a reranker orders.
Start lesson 1 →
what you learned

The map: every word, in its place.

Tokens are the addresses, vectors the content. Attention mixes them, the layers refine them, and the last step picks the next token.

WhereThe wordsLesson
The passpass, weights, LM head, inference1
Intoken, vocabulary, byte-pair encoding, merge, token ID2
Insidematrix, embedding matrix, dimension, dot product3
Insidequery, key, value, attention weights, head, causal mask4
Insidelayer, feed-forward, residual stream, parameters5
Outlogit, softmax, temperature, top-p6
Trainingloss, backpropagation, gradient descent, base model, fine-tuning, preference training, reasoning training7
Runninggraphics card, quantization, KV cache, prompt caching, thinking tokens8
Kindstransformer, decoder, encoder, reranker, mixture of experts, multimodal, diffusion9