---
title: Attention — How a model works, lesson 4
description: Each token takes a share of the tokens before it through a query, a key and a value, with three tokens worked by hand and every number of the softmax shown.
url: https://instibrain.ai/guides/how-a-model-works/attention
updated: 2026-11-08
site: https://instibrain.ai/llms.txt
---

lesson 4 of 9 · how a model works

# Attention: each token takes from the tokens that matter to it.

In attention, each token looks back at the tokens before it and takes a share of each. The shares come from dot products and add up to 1.

## new words

**Query**
What a token looks for: its vector times one weight matrix, W_Q.

**Key**
What a token offers to be found by: its vector times W_K.

**Value**
What a token hands over when it is chosen: its vector times W_V.

**Attention weights**
The shares, from 0 to 1, one token takes from each other token. They are made for this text and thrown away.

**Head**
One attention with its own three matrices. Llama 3 8B runs 32 side by side in every layer.

## the lesson

A token's vector from lesson 3 is its dictionary meaning: you has the same row in every sentence. Attention turns it into its meaning in this sentence.

An example with three tokens and vectors of two numbers, so every step fits on the page. The token you looks back at how, are, and itself. Its query meets each key in a dot product:

```
query of you = [1, 2]
key of how   = [1, 0]   1×1 + 2×0 = 1
key of are   = [0, 1]   1×0 + 2×1 = 2
key of you   = [1, 1]   1×1 + 2×1 = 3
```

Each score is divided by √2, the square root of how many numbers a key holds, so long vectors do not swamp the result. Softmax then turns the scores into shares that add up to 1: raise e, 2.718, to each score, and divide each by the sum.

```
how     are     you
÷ √2      0.707   1.414   2.121
e to it   2.028   4.113   8.342   sum 14.483
share     0.14    0.28    0.58
```

The new you is the three values, each times its share, added up. It is now the you of a how-are-you greeting. In a real model the result goes through one more matrix, W_O, and is added onto the token's own vector, so the token keeps what it was.

A chat model looks only back. A token cannot see the ones after it, because when it was written they did not exist yet: that rule is the causal mask. 1,000 tokens make 500,500 such pairs, in every head of every layer, so twice the text costs about four times the attention.

Two things are called weights. Attention weights are the shares above, made for this text and thrown away. Model weights are the numbers in W_Q, W_K, W_V and W_O, set by training. When someone says weights, ask which one.

The example's shares. You takes 0.58 from itself, 0.28 from are, and 0.14 from how; a thicker line is a bigger share. The vectors are examples.

## try it

Type this and press Enter:

```
In the sentence 'The invoice arrived late because it was sent to the old address', which word does 'it' point to, and which other words did you need to decide?
```

## you should see

- it is the invoice.
- Words such as sent and address: an invoice is a thing that is sent to an address. Following a word back to the word it means is the kind of link attention is for.

---

instibrain is made by WebWork AI, Inc. Everyone uses the same models · the difference is what yours knows.