Ingestion as a job.
Ingestion is a job: read the files, cut, embed, store each chunk with its source and who may read it. It runs again whenever a file changes.
new words
- Ingestion
- The run that turns files into the index. AI basics, lesson 6 named it; this lesson runs it.
- Source
- What is stored beside a chunk: the file, the page, the section, the date it last changed.
- Readers
- Who may read the file the chunk came from. Stored with the chunk, so a search returns nothing its asker may not read.
the lesson
The job has four steps, and it is a job, not a one-time import: it runs on a schedule, nightly at least, and after every change it is told of.
| Step | What it does | What it stores |
|---|---|---|
| Read | Opens each file it has not seen, or that changed since its last run | The file's path and its change date |
| Cut | Cuts the file into chunks by its structure, lesson 3 | The chunk's text, its section, its page number |
| Embed | Sends each chunk to the embedding model | The vector |
| Store | Writes the row | All of it, with who may read the file |
Readers matter as much as text. A search runs as a person, and a chunk from a file that person may not open never reaches the prompt. The filter runs before any vector is compared, so it costs nothing and leaks nothing.
The first run is the big one: 50 million tokens at $0.10 per million is $5. A rerun embeds only what changed: one 20-page document, 10,000 tokens, is $0.001. A file that is deleted takes its chunks with it on the next run, and an answer from last year's version means the job did not run.
try it
Type this and press Enter, with a page in a space you read:
Show me what an index would store for <page>: its first chunk's text, the file, the section, the date it last changed, and who may read it.you should see
- The first section's text, the page's path, its heading, its last change from the brain, and the people the space is shared with.
- The readers line is the one most indexes forget.