---
title: Layers and weights — How a model works, lesson 5
description: A layer is attention then feed-forward, and Llama 3 8B stacks 32 of them: count its 8,030,261,248 weights matrix by matrix, and see where most of them sit.
url: https://instibrain.ai/guides/how-a-model-works/layers-and-weights
updated: 2026-11-08
site: https://instibrain.ai/llms.txt
---

lesson 5 of 9 · how a model works

# Layers and weights: count the 8 billion.

A layer is attention, then feed-forward. A model stacks the same layer many times, and its size is the count of numbers in all their matrices.

## new words

**Layer**
Attention, then feed-forward, each adding its result onto the token's vector. Also called a transformer block.

**Feed-forward**
Each token's vector alone, through wide matrices and back. Most of a model's weights are here.

**Residual stream**
The token's vector as it passes from layer to layer, each layer adding its change.

**Parameters**
Another word for the model weights. 8B in a model's name means 8 billion of them.

## the lesson

Attention lets the tokens take from each other. Feed-forward then works on each token alone: its 4,096 numbers go up to 14,336, through a gate, and back down to 4,096. Each step adds its result onto the token's vector, so the token keeps what it was and gains the change.

Llama 3 8B stacks 32 such layers. Its maker publishes the shape, so its weights can be counted:

| Part | Shape | Weights |
| W_Q and W_O | 4,096 × 4,096, two | 33,554,432 |
| W_K and W_V | 4,096 × 1,024, two | 8,388,608 |
| Feed-forward | 4,096 × 14,336, three | 176,160,768 |
| One layer |  | 218,103,808 |
| 32 layers |  | 6,979,321,856 |
| Embedding matrix and LM head | 128,256 × 4,096, two | 1,050,673,152 |
| Norms, small scales | 65 × 4,096 | 266,240 |
| The model |  | 8,030,261,248 |

W_K and W_V are narrower because Llama 3 shares each key and value among four of its 32 heads, which saves memory when it runs.

Feed-forward holds 176 of the 218 million weights of a layer: 81%. Studies that look inside open models find much of what a model knows stored there. They also find that early layers settle how pieces form words, middle layers hold meaning, and late layers choose what comes next. Nobody assigned those roles; training produced them.

Llama 3 70B has 80 layers of 8,192 numbers: 70,553,706,496 weights by the same count. A model's size is only this: how many layers, and how wide.

One layer and where its weights sit. The model is 32 of them, then the LM head.

## try it

Type this and press Enter:

```
Compute 32 × (2 × 4096 × 4096 + 2 × 4096 × 1024 + 3 × 4096 × 14336) + 2 × 128256 × 4096. Say the result in billions.
```

## you should see

- 8,029,995,008: about 8.03 billion.
- The 266,240 norm weights in the table make up the rest of the published 8,030,261,248.

---

instibrain is made by WebWork AI, Inc. Everyone uses the same models · the difference is what yours knows.