Layers and weights: count the 8 billion.
A layer is attention, then feed-forward. A model stacks the same layer many times, and its size is the count of numbers in all their matrices.
new words
- Layer
- Attention, then feed-forward, each adding its result onto the token's vector. Also called a transformer block.
- Feed-forward
- Each token's vector alone, through wide matrices and back. Most of a model's weights are here.
- Residual stream
- The token's vector as it passes from layer to layer, each layer adding its change.
- Parameters
- Another word for the model weights. 8B in a model's name means 8 billion of them.
the lesson
Attention lets the tokens take from each other. Feed-forward then works on each token alone: its 4,096 numbers go up to 14,336, through a gate, and back down to 4,096. Each step adds its result onto the token's vector, so the token keeps what it was and gains the change.
Llama 3 8B stacks 32 such layers. Its maker publishes the shape, so its weights can be counted:
| Part | Shape | Weights |
|---|---|---|
| W_Q and W_O | 4,096 × 4,096, two | 33,554,432 |
| W_K and W_V | 4,096 × 1,024, two | 8,388,608 |
| Feed-forward | 4,096 × 14,336, three | 176,160,768 |
| One layer | 218,103,808 | |
| 32 layers | 6,979,321,856 | |
| Embedding matrix and LM head | 128,256 × 4,096, two | 1,050,673,152 |
| Norms, small scales | 65 × 4,096 | 266,240 |
| The model | 8,030,261,248 |
W_K and W_V are narrower because Llama 3 shares each key and value among four of its 32 heads, which saves memory when it runs.
Feed-forward holds 176 of the 218 million weights of a layer: 81%. Studies that look inside open models find much of what a model knows stored there. They also find that early layers settle how pieces form words, middle layers hold meaning, and late layers choose what comes next. Nobody assigned those roles; training produced them.
Llama 3 70B has 80 layers of 8,192 numbers: 70,553,706,496 weights by the same count. A model's size is only this: how many layers, and how wide.
try it
Type this and press Enter:
Compute 32 × (2 × 4096 × 4096 + 2 × 4096 × 1024 + 3 × 4096 × 14336) + 2 × 128256 × 4096. Say the result in billions.you should see
- 8,029,995,008: about 8.03 billion.
- The 266,240 norm weights in the table make up the rest of the published 8,030,261,248.