instibrain
Sign in
instibrain  /  guides  /  rag

Retrieval beside your brain, in eight short lessons.

AI basics said what a vector, an embedding and a chunk are. These lessons run a retrieval pipeline beside the brain: what goes in it, the job that builds it, the cuts, hybrid search, the reranker, the prompt, the measure that says which step failed, and where your pages sit.

Lesson 1: What goes in the index → AI basics comes first

Every number is computed on the page or is a list price. Lesson 3 has a live toy that cuts a page.

the lessons · the words under it

Eight lessons, from the files to the loop back to a page.

The first two decide what is indexed and how the job runs. Lessons 3 to 6 are the four steps, each with its number. Lesson 7 measures them, and lesson 8 puts the pages beside the index.

  1. 1What goes in the indexA reviewed position is a page, read whole; thousands of raw files nobody reviewed are the index.
  2. 2Ingestion as a jobA job reads the files, cuts, embeds and stores, with each chunk's source and who may read it; it runs again on change.
  3. 3Chunking by structureCut on headings and paragraphs, carry the section's title, repeat one sentence as overlap; the toy cuts a page live.
  4. 4Hybrid searchKeyword search beside vector search, the two rankings merged, so a code and a meaning are both found.
  5. 5The rerankerA second model reads the question with each of the top chunks and reorders them; the best five go in.
  6. 6The prompt and the citation rulePassages first, numbered with their source, the question last, and the rule: answer only from them, or say so.
  7. 7Measuring recallTwenty real questions, the chunk that answers each, and the share found in the top five; under 80%, fix retrieval.
  8. 8Where instibrain sitsPages read whole beside the index, and the loop that turns an answer found in the files into a page.
Start lesson 1 →
what you learned

The numbers to hold.

Each is computed in its lesson, from the counts given there and Claude's list prices.

NumberWhat it is
200,000 tokens400 pages at 500 tokens: a fifth of the window, read whole.
50,000,000 tokens5,000 documents of 20 pages: fifty windows, so an index.
$5, then $0.001The first embedding run at $0.10 per million, then one changed 20-page document.
150 to 600 wordsOne chunk, 200 to 800 tokens, cut on structure.
1 / (60 + rank)Rank fusion: a chunk's score in each list, added up.
26,000 tokensWhat a reranker reads for 50 chunks of 500 and a 20-token question.
$0.0053 + $0.0015One question on Sonnet 5.5: 2,630 tokens in, 150 out.
80%Recall at 5 on twenty questions, the line under which retrieval is fixed first.