Running a model: memory decides the speed.
To write one token, the model reads every weight once. So the memory that holds the weights, and how fast it reads them, decide the speed.
new words
- Graphics card, GPU
- The chip that runs a model, with its own fast memory.
- Quantization
- Storing each weight in fewer bits, 8 or 4 in place of 16. Smaller and faster, with a small loss of quality.
- KV cache
- The keys and values of every token so far, kept so the next pass computes only the new token's.
- Thinking tokens
- Tokens a reasoning model writes before its answer. Each one is a full pass, and you pay for it.
the lesson
The weights must sit in memory, and every pass reads all of them. Memory is weights × bytes per weight:
| Stored at | Bytes each | Llama 3 8B | Llama 3 70B |
|---|---|---|---|
| 16 bits | 2 | 16.1 GB | 141 GB |
| 8 bits | 1 | 8.0 GB | 70.6 GB |
| 4 bits | 0.5 | 4.0 GB | 35.3 GB |
One H100, a graphics card common for this work, holds 80 GB and reads 3,350 GB a second, by its maker's figures. One pass of the 8B model at 16 bits takes 16.1 / 3,350 = 0.0048 seconds: at most 208 tokens a second. From a laptop's memory, at an example 100 GB a second, it takes 0.16 seconds: 6 a second. The arithmetic is cheap; moving the numbers is the cost.
The 70B model at 16 bits needs 141 GB, so two such cards. At 4 bits it fits on one.
The KV cache keeps the keys and values of lesson 4 for every token, so a new pass does not compute them again. For Llama 3 8B that is 2 × 32 layers × 1,024 numbers × 2 bytes = 131,072 bytes per token. A conversation of 100,000 tokens needs 13.1 GB of cache, nearly the size of the model. A long context costs memory.
A provider keeps the cache of a beginning it has already read, so the next call that starts the same way pays less for it: prompt caching, with Claude's prices in AI basics, lesson 1. A reasoning model writes its working first, in thinking tokens; each is one more full pass, so thinking is slower and costs more. In Claude Code the knob is effort: Claude basics, lesson 3.
try it
Type this and press Enter:
A model has 70 billion weights. How many gigabytes does it need at 16, 8 and 4 bits per weight, and which of them fit on one 80 GB graphics card?you should see
- 140, 70 and 35 GB.
- 8 and 4 bits fit on one 80 GB card, and 16 bits needs two. The answer may add that the running conversation needs room too.