---
title: Chunking and the index — AI basics, lesson 6
description: A document is cut into chunks of 200 to 800 tokens, each chunk gets one vector, and the table of them is the index: the counts, cost and size of a firm's files.
url: https://instibrain.ai/guides/ai-basics/chunking-and-the-index
updated: 2026-11-07
site: https://instibrain.ai/llms.txt
---

lesson 6 of 8 · ai basics

# Chunking and the index.

A document is cut into chunks of 200 to 800 tokens. Each chunk gets one vector. The table of chunks and vectors is the index.

## new words

**Chunk**
One slice of a document. The unit that is embedded, stored and retrieved.

**Overlap**
The last sentences of one chunk repeated at the start of the next, so a fact on the border is whole in one of them.

**Index**
The stored table: each chunk's text, its vector, and where it came from: file, page, section, date.

**Vector database**
Software that keeps the index and finds the nearest vectors fast.

**Ingestion**
The run that reads the files, chunks, embeds and stores. Repeated when files change.

## the lesson

A 40-page contract is about 20,000 tokens. At 500-token chunks that is 40 chunks; with 50 tokens of overlap, 44.

A firm with 5,000 documents of 20 pages holds 50 million tokens:

| Step | Figure | How |
| Chunks | 100,000 | 50M tokens ÷ 500 |
| Embedding cost, once | $5 | 50M tokens × $0.10 per million |
| Storage | 614 MB | 100,000 × 1,536 numbers × 4 bytes |
| Search, brute force | under 0.1 s | 100,000 × 1,536 multiplications per question |
| Fast index needed from | ≈ 1M chunks | HNSW, the method inside vector databases |

Cut on headings and paragraphs, never every N characters. A chunk must make sense alone, without its neighbours. Store the source with every chunk; a filter on file type or date does half the work before any vector is compared.

One page cut into three chunks with overlap, each embedded, each stored with its source. The values are examples.

## try it

Type this and press Enter:

```
Take the longest page in a space I read. Cut it into pieces of one to three paragraphs. For each piece, say whether it still says whose, when and what when read alone.
```

## you should see

- A piece that cannot be understood alone is too small. A piece that covers two topics is too big.
- Most pieces land between 150 and 600 words, which is 200 to 800 tokens.

---

instibrain is made by WebWork AI, Inc. Everyone uses the same models · the difference is what yours knows.