Agent Lore

Why output changes when you didn't change anything

Your prompt is the same. Your code is the same. The output is different. Legitimate causes:

• The provider updated the model behind an alias you pinned loosely
• You upgraded a tool or SDK that alters the system prompt
• Your CLAUDE.md grew and pushed something important out of attention
• A skill's description started matching requests it shouldn't
• Temperature, or genuine sampling variance

Most teams find out from a user. A regression set finds out first.

The set

Different from a golden set. A golden set asks *is this correct?* A regression set asks *is this the same as last time?* Fewer cases, tighter checks, higher frequency.

[
  {
    "id": "commit-message-format",
    "prompt": "Write a commit message for this diff",
    "fixture": "fixtures/small-refactor.diff",
    "assert": {
      "matches": "^(feat|fix|chore|docs|refactor)\\([a-z-]+\\): .{10,72}$",
      "max_output_tokens": 120
    }
  },
  {
    "id": "refuses-without-context",
    "prompt": "What does the deploy script do?",
    "fixture": "fixtures/empty-repo/",
    "assert": {
      "contains_any": ["could not find", "no deploy script", "does not exist"],
      "not_contains": ["typically", "usually", "generally"]
    }
  }
]

That second case is worth dwelling on. It asserts the agent says *"I couldn't find it"* rather than describing what a deploy script usually does. Hedging words like "typically" are a reliable signal that the model has stopped reading your repo and started reciting. Catching that drift early is most of the value.

Assertions that hold up

Structural, not semantic. Assert the format, the length, the presence of a citation, not the exact prose. Prose varies run to run without anything being wrong.

Good:  output matches /^(feat|fix|chore)\(.+\): /
Good:  output cites at least one file:line
Good:  output is under 200 tokens
Bad:   output equals "fix(parser): handle empty input"

Negative assertions catch the most. What should *never* appear is often sharper than what should:

not_contains: ["As an AI", "I don't have access", "typically", "in general"]

Pin the model exactly. claude-haiku-4-5-20251001, not claude-haiku-latest. An alias that silently moves is the drift you are trying to detect.

Wiring it up

name: prompt-regression
on:
  pull_request:
    paths: ['prompts/**', 'CLAUDE.md', '.claude/skills/**']
  schedule:
    - cron: '0 6 * * 1'   # weekly, to catch provider-side drift

jobs:
  regress:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python3 evals/regress.py --set regression.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Two triggers matter and they catch different things. The path filter catches your changes. The schedule catches everyone else's.

When it fires

Do not immediately fix the prompt. First establish which kind of change it was:

1. Re-run. Once is variance; three times is a change.
2. Diff the inputs. Did CLAUDE.md grow? Did a skill get installed? Check git.
3. Pin harder. If the model alias moved, that is your answer.
4. Only then adjust the prompt, and add the failing case to the golden set.

See also: Build a Golden Task Set · Write an Effective CLAUDE.md