Retrieval or Just Put It In Context?

The decision changed

When windows were 8K tokens, retrieval was mandatory. At 1M tokens, stuffing everything in is often simpler, more accurate, and, accounting for engineering time, cheaper.

But not always, and the boundary is worth knowing.

Just put it in context when…

The corpus fits with room to spare. Under ~100K tokens against a 1M window, retrieval is machinery you do not need.
The question needs the whole thing. "Is this API design consistent?" cannot be answered from three retrieved chunks. Neither can "what changed between these two versions?"
Relationships between distant parts matter. Retrieval fetches locally relevant pieces and severs the connections between them.
The corpus is stable and the queries repeat. Cache the whole thing as a prefix and each query costs a tenth of the input price. This is often cheaper *and* better than retrieval.

Retrieve when…

The corpus does not fit. Millions of tokens. No window solves this.
Only a tiny slice is ever relevant. One support article out of fifty thousand. Sending all fifty thousand to answer one question is enormously wasteful.
Freshness matters per-query. Content changing hourly cannot be a cached prefix.
Volume is high and cost dominates. At a million queries, the difference between 2,000 retrieved tokens and 200,000 stuffed tokens is the whole budget.

The arithmetic

50,000 queries a month against a 200,000-token corpus, on a model at $2/M input:

Stuffed, no caching:   200,000 × 50,000 × $2/M  = $20,000
Stuffed, cached:       ~10% of input             ≈  $2,000
Retrieved (~3k/query):   3,000 × 50,000 × $2/M  =    $300

Retrieval wins by a lot here. Now the same corpus at 500 queries a month:

Stuffed, cached:  $20
Retrieved:        $3   (plus an embedding pipeline you now maintain)

Seventeen dollars is not worth a vector database. Query volume, not corpus size, is what usually decides.

The hybrid worth knowing

Retrieve generously, then let the model read properly. Instead of 5 tight chunks, pull 30 loose ones and give the model the surrounding file when it asks.

1. Embed the query, pull 30 candidates
2. Put all 30 in context (long windows make this affordable)
3. Give the agent a read_file tool so it can pull full context on
   anything that looks relevant

This gets retrieval's cost profile with much of stuffing's accuracy, and it degrades gracefully when the embedding misses. The agent can go and look.

Start simple

Put it in context. Measure. Add retrieval when the numbers say to, not because retrieval is what serious systems do. Most corpora people build pipelines for would fit in a cached prefix.