lesson 6 of 8 · ai basics
Chunking and the index.
A document is cut into chunks of 200 to 800 tokens. Each chunk gets one vector. The table of chunks and vectors is the index.
new words
- Chunk
- One slice of a document. The unit that is embedded, stored and retrieved.
- Overlap
- The last sentences of one chunk repeated at the start of the next, so a fact on the border is whole in one of them.
- Index
- The stored table: each chunk's text, its vector, and where it came from: file, page, section, date.
- Vector database
- Software that keeps the index and finds the nearest vectors fast.
- Ingestion
- The run that reads the files, chunks, embeds and stores. Repeated when files change.
the lesson
A 40-page contract is about 20,000 tokens. At 500-token chunks that is 40 chunks; with 50 tokens of overlap, 44.
A firm with 5,000 documents of 20 pages holds 50 million tokens:
| Step | Figure | How |
|---|---|---|
| Chunks | 100,000 | 50M tokens ÷ 500 |
| Embedding cost, once | $5 | 50M tokens × $0.10 per million |
| Storage | 614 MB | 100,000 × 1,536 numbers × 4 bytes |
| Search, brute force | under 0.1 s | 100,000 × 1,536 multiplications per question |
| Fast index needed from | ≈ 1M chunks | HNSW, the method inside vector databases |
Cut on headings and paragraphs, never every N characters. A chunk must make sense alone, without its neighbours. Store the source with every chunk; a filter on file type or date does half the work before any vector is compared.
try it
Type this and press Enter:
Take the longest page in a space I read. Cut it into pieces of one to three paragraphs. For each piece, say whether it still says whose, when and what when read alone.you should see
- A piece that cannot be understood alone is too small. A piece that covers two topics is too big.
- Most pieces land between 150 and 600 words, which is 200 to 800 tokens.