---
title: Evals and the golden set — Around the model, lesson 7
description: A benchmark tests the model; only your own questions test your system. Build a golden set, score five kinds of answer, run it after every change, and judge each answer live.
url: https://instibrain.ai/guides/around-the-model/evals-and-the-golden-set
updated: 2026-11-08
site: https://instibrain.ai/llms.txt
---

lesson 7 of 9 · around the model

# A model always answers. An eval says when it is right.

A benchmark tells you about the model. Only your own questions, each with its right answer, tell you about your system: the model plus the context you build for it.

## new words

**Benchmark**
A public test, such as exam questions or coding tasks, that ranks models against each other.

**Golden set**
Real questions, each with its right answer or "none", marked by people. Kept, and run again after every change.

**Abstention**
The AI says it does not know. Right when the answer is not there; wrong when it is.

**Regression**
A change after which questions that passed before fail.

**Live judge**
A second model that checks each answer against its sources before anyone sees it.

## the lesson

A benchmark runs the model alone. Your system is the model plus the context you build for it, so only your own questions test it.

A golden set, in a week. Take 150 real questions from past tickets. For each, the team marks the document that answers it, or "none". First run the search alone, and count how often that document is in the top five: recall at 5 (RAG, lesson 7). Then run the whole system, and mark each answer:

| The answer is there | The AI | Name |
| Yes | answers right | correct |
| Yes | answers wrong | wrong answer |
| Yes | says it does not know | false abstention |
| No | says it does not know | correct abstention |
| No | answers anyway | hallucination: the worst cell |

Sort the wrong ones: the search found the wrong text, or the model misread the right one. Run the same 150 after every change. A drop is a regression, and the change goes back.

That is an offline eval: a fixed set, before launch. An online eval compares live answers with what people did after. A live judge works in between: a second model reads each answer beside the text it rests on, and stops an answer that text does not support before anyone sees it. The test it applies is grounding: an answer says only what its text supports.

The same test, run by people, is the golden set of an instibrain implementation: before the work, the team writes twenty real questions and their right answers; its AI answers them before and after; its people score the answers, and run the same questions again as the work changes (enterprise).

Each answer of an eval lands in one of five cells. A hallucination, an answer where the text holds none, is the worst.

## try it

Write five questions you ask at work each week, with the right answer to each. Then ask each one in a new Claude window, like this:

```
When are supplier invoices paid? Answer only from the files you can read, and name the file. If they do not say, say you do not know.
```

## you should see

- Each answer lands in one of the five cells.
- Your count, such as 3 correct of 5, is your first eval. Ask the same five again after you change a file.

---

instibrain is made by WebWork AI, Inc. Everyone uses the same models · the difference is what yours knows.
