Kinds of models: one design, three last steps.
Nearly every model that reads or writes language is one design, the transformer. What differs is its last step and its training.
new words
- Transformer
- The design of lessons 3 to 6: an embedding matrix, layers of attention and feed-forward, and a last step. Published in 2017.
- Decoder
- A transformer that looks only back and writes the next token. Every chat model is one.
- Encoder
- A transformer that reads the whole text in both directions at once, to understand it, not to continue it.
- Mixture of experts
- A model whose feed-forward is split into experts, of which only a few run for each token.
the lesson
| Kind | In | Out | For |
|---|---|---|---|
| Chat model, a decoder | text | the next token, again and again | writing answers; Claude is one |
| Embedding model, an encoder | one text | one vector | search by meaning |
| Reranker, an encoder | two texts together | one score | putting the best results first |
An embedding model is trained on pairs of texts, so texts that mean the same land close: AI basics, lesson 4. A reranker reads the question and one passage together, which is slower and sharper than comparing two vectors: RAG, lesson 5. A retrieval system runs them in a row: the embedding model finds the chunks, and the chat model writes from them. An index belongs to its embedding model; vectors from two models cannot be compared.
| Variation | What changes |
|---|---|
| Reasoning model | Reasoning training, lesson 7: it writes its working first, in thinking tokens. |
| Mixture of experts | A router picks a few experts for each token. Fewer weights run per token, and every expert still sits in memory. |
| Multimodal | An image is cut into small squares, and each becomes a vector in the same row as the text's tokens. A 224 × 224 image in 16 × 16 squares is 14 × 14 = 196 of them. |
Not every model is a transformer. Most image generators are diffusion models, which turn noise into a picture step by step. A bank's fraud score is often classical machine learning, such as decision trees, with no layers at all. What reads or writes language today is nearly always a transformer.
try it
Type this and press Enter:
Which kind of model are you? Which kind would find, among 10,000 files, the ones that answer a question by meaning, and which would put the best five in order?you should see
- A chat model: a decoder that writes one token at a time.
- An embedding model to find the candidates by meaning, and a reranker to put the best few in order.