Agent Lore

The shape of the problem

An agentic loop is wasteful in one very specific way: every turn re-sends everything that came before. Your system prompt, your tool definitions, the file you read four turns ago, the whole conversation. Turn 30 might send 60,000 input tokens of which 58,000 are byte-for-byte identical to turn 29.

Prompt caching bills that repeated prefix at a fraction of the normal rate.

The multipliers

Prices differ by provider, but the structure is consistent:

A hit costs a tenth of fresh input. That is the whole game.

When it pays off

Because a 5-minute write costs 1.25× and a hit saves 0.9×, the write pays for itself after one hit:

write cost:  1.25 × base
saving/hit:  0.90 × base
break-even:  1.25 / 0.90 ≈ 1.4 hits

For the 1-hour cache at 2×, break-even is a bit over two hits. In an agentic session you will get dozens.

Getting hits

Caching only works on an exact prefix match. The rules that follow from that:

1. Put stable content first. System prompt, tool definitions, and long reference documents go at the top. Anything that changes per-turn goes at the bottom.
2. Do not interpolate volatile values into the prefix. A timestamp, a random id, or a "current time" line at the top of your system prompt invalidates the entire cache on every single call. This is the most common cause of a 0% hit rate.
3. Watch the 5-minute window. The cache expires on idle. A developer thinking for ten minutes between prompts pays the write cost again. That is fine. It is still cheaper than not caching.
4. Batch your reads. Reading six files in one turn caches them together. Reading them across six turns writes the cache six times.

Checking your hit rate

The API reports it per call:

{
  "usage": {
    "input_tokens": 9012,
    "cache_read_input_tokens": 36104,
    "cache_creation_input_tokens": 0,
    "output_tokens": 1180
  }
}

Hit rate is cache_read / (cache_read + input_tokens). Here that is 80%.

If you see cache_creation high and cache_read near zero on every call, something in your prefix is changing. Find it. It is almost always a timestamp.

When caching does not help

• Every call is genuinely different. Bulk classification over unrelated documents shares only the system prompt. Cache that, expect a modest saving, and reach for the Batch API instead.
• Your prefix is tiny. Below roughly a thousand tokens the bookkeeping is not worth it, and some providers set a minimum.

See also: Keep AI Costs Low · The Codex