Verify What the Agent Tells You

The failure this prevents

An agent reports: *"I've updated the config and all tests pass."*

Three things could be true. It did the work and verified it. It did the work and assumed the tests pass. It did something adjacent and described what it intended.

The output looks identical in all three cases. Confidence is not evidence, and fluent text is not a receipt.

Make the agent produce receipts

The cheapest intervention is a standing instruction. In CLAUDE.md:

## Reporting rules

- Never claim a test passes without pasting the command and its output.
- Cite file:line for every claim about this codebase.
- If you could not verify something, say "unverified" explicitly.
- "Should work" is not an acceptable report. Run it.

This costs a few dozen tokens per request and changes the failure mode from silent to visible.

The four checks, cheapest first

1. Does it cite? A claim about your code should carry file:line. If it doesn't, the model is probably recalling a pattern rather than reading your repo. Ask for the citation. The answer often changes.

2. Does the command actually run? For anything the agent claims works:

git diff --stat          # did it change what it said it changed?
npm test                 # do the tests actually pass?

Ten seconds. Catches the majority of confident-but-wrong reports.

3. Does the negative case hold? Agents are good at making a test pass and bad at noticing they made it pass trivially.

# Break the thing on purpose. The test must fail.
git stash && npm test    # expect failure
git stash pop && npm test # expect pass

A test that passes both ways is testing nothing. This catches the single most common form of fake work.

4. Is it internally consistent? Ask the same question a different way in a fresh session. Confident answers that change between phrasings were never grounded.

Automate the check into the loop

The strongest version is not checking afterwards. It is making the agent unable to finish without checking. That is what a validator in a skill does:

After generating the migration, run scripts/validate-migration.sh.
If it exits non-zero, fix it and run again.
Do not report success until it exits zero. Paste the final output.

See The Armory for the full pattern.

Where to spend the effort

You cannot verify everything, and you should not try. Scale the check to the blast radius:

Change	Check
A comment, a doc	Read it
A function	Run the tests
A migration, a config, anything in CI	Run it, roll it back, run it again
Anything touching auth, payments, or deletion	Read every line yourself

The agent is a fast, confident colleague who does not know when it is wrong. Treat its output like a pull request from someone talented and new.