Agent Lore

The problem it solves

You tweak a system prompt. It feels better. Is it? You switch to a cheaper model. Did quality drop, or did you just get unlucky on the two examples you tried?

Without a fixed set of cases you are guessing, and guessing is how people end up paying frontier-tier prices for work a fast-tier model handled fine.

What a golden set is

Thirty to fifty cases, each with an input and a known-good outcome. Small enough to run in a couple of minutes. Fixed, so results are comparable across runs.

[
  {
    "id": "extract-invoice-total",
    "input": "invoices/2026-03-acme.pdf",
    "expect": { "total": 4820.50, "currency": "USD" },
    "check": "exact"
  },
  {
    "id": "refactor-preserves-behaviour",
    "input": "fixtures/legacy-parser.js",
    "expect": "tests/parser.test.js passes",
    "check": "command"
  },
  {
    "id": "flags-sql-injection",
    "input": "fixtures/vulnerable-query.py",
    "expect": ["sql injection", "parameterised"],
    "check": "contains-any"
  }
]

Choosing the cases

The instinct is to pick representative examples. Do that for a third of the set, then deliberately fill the rest with the cases that actually discriminate:

• Ones that have failed before. Every production bug becomes a case. This is the highest-value source and it costs nothing to collect.
• Edge cases. Empty input, enormous input, malformed input, ambiguous input.
• Near-misses. Two cases that look similar but need different answers. These are what separate a model that understands from one that pattern-matches.
• Cases where the right answer is "I don't know". Models are bad at this and it matters enormously for agentic work. Include a few where the correct behaviour is refusing to guess.

A set of thirty easy cases that every model passes tells you nothing. You want a set where the fast tier fails a few.

The three check types

Exact: deterministic output. Extraction, classification, structured data. Cheap and unambiguous.

Command: the output is code, and the check is whether it works.

npm test -- parser.test.js

This is the strongest check available. Where you can express the expectation as a passing command, do.

Contains: the output is prose and you are checking that key points appear. Weakest and the most prone to false passes, but sometimes the only option.

Running it

# Run the set against each tier and compare
for model in claude-haiku-4-5 claude-sonnet-5 claude-opus-5; do
  python3 evals/run.py --model "$model" --set golden.json --out "results-$model.json"
done

python3 evals/compare.py results-*.json

What you want out the other end:

                     passed   cost      p50 latency
claude-haiku-4-5      41/50   $0.11     1.2s
claude-sonnet-5       48/50   $0.34     2.1s
claude-opus-5         49/50   $0.88     3.8s

That table is the entire model-selection decision, made once, with evidence. Here Sonnet is the answer: Opus buys one more case for 2.6× the price.

The discipline

• Freeze the set. Changing cases and prompts at the same time tells you nothing. Change one thing per run.
• Version it. golden-v3.json in the repo, alongside the results.
• Run it in CI on prompt changes. A prompt is code. Treat a quality regression like a failing test.
• Grow it from failures, not from imagination. Every real-world miss becomes case 51.

See also: Regression Prompts · Verify What the Agent Tells You · Choose the Right Model