lesson 4 of 8 · ai basics
An embedding is a vector made from text.
An embedding model reads a text and writes a vector. Two texts that mean the same get nearby vectors, even with no word in common.
new words
- Embedding
- The vector a model writes for one text. 768 to 3,072 numbers with today's models.
- Embedding model
- A small model trained only to write those vectors. It is not the chat model.
- Semantic
- About meaning. A semantic match is "bill settled" matching "invoice paid".
- Lexical
- About the words themselves. A lexical match needs the same word in both texts.
the lesson
The simplest text vector you can build by hand counts words. Fix a word list: invoice, paid, day, bill, settled, month. Each text becomes six counts.
| Text | invoice | paid | day | bill | settled | month |
|---|---|---|---|---|---|---|
| Invoices are paid within 30 days | 1 | 1 | 1 | 0 | 0 | 0 |
| The bill is settled in a month | 0 | 0 | 0 | 1 | 1 | 1 |
The two sentences say the same thing and share no column. Counting words fails on synonyms, and most questions use other words than the document.
An embedding model was trained on billions of text pairs to put "bill settled" next to "invoice paid". It chose its own columns: column 412 of a 1,536-number vector has no name, and needs none. What you get:
| Question | Word counts | Embedding model |
|---|---|---|
| Synonyms match | no | yes |
| Another language matches | no | yes, with a multilingual model |
| Exact codes and names match | yes | weakly; hybrid search fixes it (lesson 7) |
| Cost per million tokens | 0 | $0.02 to $0.20, the list prices by model |
| A 300-page book, 150,000 tokens | 0 | 3 cents at most |
try it
Type this and press Enter:
Take one sentence from a page you read for me, and write the question a colleague would type to find it. Then list the words the two have in common, ignoring the, is, a and of.you should see
- Zero or one shared word, most of the time. That gap is where keyword search fails and an embedding does not.