---
title: Running a model — How a model works, lesson 8
description: Every token reads every weight once, so memory decides speed: an 8B model needs 16 GB at 16 bits and 4 GB at 4, and a long context fills the KV cache.
url: https://instibrain.ai/guides/how-a-model-works/running-a-model
updated: 2026-11-08
site: https://instibrain.ai/llms.txt
---

lesson 8 of 9 · how a model works

# Running a model: memory decides the speed.

To write one token, the model reads every weight once. So the memory that holds the weights, and how fast it reads them, decide the speed.

## new words

**Graphics card, GPU**
The chip that runs a model, with its own fast memory.

**Quantization**
Storing each weight in fewer bits, 8 or 4 in place of 16. Smaller and faster, with a small loss of quality.

**KV cache**
The keys and values of every token so far, kept so the next pass computes only the new token's.

**Thinking tokens**
Tokens a reasoning model writes before its answer. Each one is a full pass, and you pay for it.

## the lesson

The weights must sit in memory, and every pass reads all of them. Memory is weights × bytes per weight:

| Stored at | Bytes each | Llama 3 8B | Llama 3 70B |
| 16 bits | 2 | 16.1 GB | 141 GB |
| 8 bits | 1 | 8.0 GB | 70.6 GB |
| 4 bits | 0.5 | 4.0 GB | 35.3 GB |

One H100, a graphics card common for this work, holds 80 GB and reads 3,350 GB a second, by its maker's figures. One pass of the 8B model at 16 bits takes 16.1 / 3,350 = 0.0048 seconds: at most 208 tokens a second. From a laptop's memory, at an example 100 GB a second, it takes 0.16 seconds: 6 a second. The arithmetic is cheap; moving the numbers is the cost.

The 70B model at 16 bits needs 141 GB, so two such cards. At 4 bits it fits on one.

The KV cache keeps the keys and values of lesson 4 for every token, so a new pass does not compute them again. For Llama 3 8B that is 2 × 32 layers × 1,024 numbers × 2 bytes = 131,072 bytes per token. A conversation of 100,000 tokens needs 13.1 GB of cache, nearly the size of the model. A long context costs memory.

A provider keeps the cache of a beginning it has already read, so the next call that starts the same way pays less for it: prompt caching, with Claude's prices in AI basics, lesson 1. A reasoning model writes its working first, in thinking tokens; each is one more full pass, so thinking is slower and costs more. In Claude Code the knob is effort: Claude basics, lesson 3.

The weights of two models at three sizes, in GB, against the memory of one 80 GB card. What crosses the line needs a second card.

## try it

Type this and press Enter:

```
A model has 70 billion weights. How many gigabytes does it need at 16, 8 and 4 bits per weight, and which of them fit on one 80 GB graphics card?
```

## you should see

- 140, 70 and 35 GB.
- 8 and 4 bits fit on one 80 GB card, and 16 bits needs two. The answer may add that the running conversation needs room too.

---

instibrain is made by WebWork AI, Inc. Everyone uses the same models · the difference is what yours knows.