When the Context Window Fills Up
Part 17: What a retrieval loop should clear, compact and keep outside its window as its own evidence fills it, with implementations
Recap
In Part 16, we moved retrieval inside the model's own loop.
Instead of retrieving once and answering, the model searched, read what came back, and decided whether to search again, switch tools, answer or stop.

Part 16 measured that loop on a 40-document corpus with eight information needs. A hybrid loop that kept the full question and skipped documents it had already seen found all eight, at two calls on average.
A second policy, which added distinctive terms from the documents it had just read to its next query, never found a target. With no useful stopping rule, raising its call limit from one to eight took its average retrieved tokens from 25.5 to 200.4:

Part 16 also named the problem this part takes up, context bloat, where prefill grows and the relevant excerpts compete with duplicates and irrelevant text.
Its experiments ran without a model, so a target counted as found once it was retrieved. This part adds a model that reads the window and measures whether it can still use that evidence once the window fills.
A loop that searches across a long task keeps everything it reads, so each round adds to a finite window.

General context engineering, with its write, select, compress and isolate operations, is covered in the AI Agents course lesson below, so this part stays specific to retrieval.

We start with Part 13's effective length and why a bigger window does not solve the problem, then take apart the window of one recorded investigation.
Four ways of making room follow: clearing old tool results, compacting the history into a summary, keeping the record outside the window, and giving sub-tasks windows of their own.
The hands-on replays the same 40-step investigation under the first three, with a 1.5B model reading every window and Claude as a second writer and reader, and prices them at production scale.
The policy that did best under a budget kept the full record and searched it when a question arrived.
Why a bigger window does not fix it
When a retrieval loop overruns its window, the obvious fix is a model with a bigger one.
That moves the overflow much later, but context bloat starts long before the window is full. Besides the growing prefill, it hurts the evidence in two ways:
Pollution is text in the window that no longer helps.
Failure twoRelevance decay means the evidence that does help gets harder for the model to use as the window grows.
On the left of the figure below, the clause the loop needs competes with passages that no longer help. On the right, every passage helps, and the model still uses the same clause less well at step 30 than at step 3:

What Part 13 showed
On the NoLiMa benchmark, 13 models that all advertise at least 128K tokens of context degraded sharply as the context grew, on questions that share almost no words with the facts they ask about.
At 32K, 11 of them scored less than half of their short-context baseline.
The authors define a model's effective length as the largest tested length at which its score stays above 85 percent of its short-context score.
For every one of these models, that is far below the advertised window. Part 13 drew the idea as an illustrative curve:

Why one retrieval is safe and a loop is not
Preloading, in Parts 13 to 15, ran straight into this limit, because a preloaded corpus fills the window by design.
Retrieval mostly avoids it, although on questions like these the limit sits only a few thousand tokens in. Even GPT-4o stayed above 85 percent of its baseline only up to 8K tokens, and 10 of the 13 models sat at 2K or less.

A single retrieval mostly gets away with a limit that low because its passages match the question, often in the question's own words.
In Part 13, a fact written in the question's own words was found at every length we tried.
A loop loses that advantage as it runs. Each round adds passages that matched an earlier query, and nothing takes them out.
The loop we replay in this part reads four passages per search. With the tool call and the model's short note on the result, a step adds a median of 567 tokens.

After 40 steps, the window holds 22,653 tokens, short of the 32,768 that its model, Qwen2.5-1.5B-Instruct, accepts. In the narrow sense, this loop never overflows.
At step 14, though, it passes 8,192 tokens, the longest tested length at which this model still met NoLiMa's 85 percent rule on the loop's own passages.
Smaller models reach that line sooner. In Part 13's experiment on Qwen2.5-0.5B-Instruct, a reworded fact was missed at two of three depths with only 512 tokens of filler.
Failure one: pollution
Much of what a loop retrieves stops being useful once it has been read: passages about an abandoned lead, raw tool output whose one useful number has been extracted, and near-duplicates of passages it already holds.
All of it stays in the window, and every token takes a share of the attention budget we met in Part 13.

A relevant passage gets a thinner share as the irrelevant ones pile up.
Near-matches are the worst kind of pollution in a loop, because each passage was retrieved for resembling a query and many look like the evidence the model is after. Chroma's context rot study (July 2025) found that even a single distractor lowers accuracy.
Failure two: relevance decay
Even when nothing else in the window can distract the model, it uses the relevant tokens less well as the window grows.
In "Context Length Alone Hurts LLM Performance Despite Perfect Retrieval" (October 2025), Du and colleagues measured this drop even with the irrelevant tokens masked, so that the model could attend only to the relevant ones.
What the loop's own passages do to a small model
Experiment 1 ran the 1.5B on two kinds of text.
On routine filler, length alone barely hurt. Accuracy was 99 percent at 1K tokens and 94 percent at 32K.
On the loop's own passages, full of facts of the same kind, accuracy fell from 93 percent to 66, and 133 of the 145 wrong answers gave the value of another fact of the same kind.

The experiment does not separate length from neighbours, since on the loop's passages the two grow together.
The two failures have different fixes. Clearing junk out of the window helps with pollution, and relevance decay needs a shorter window.
Restating the evidence next to the question also helps with pollution, without removing anything.
In a pilot with the 1.5B on 150 HotpotQA questions, each needing two paragraphs joined, restating the two answering paragraphs beside the question lifted accuracy from 40 to 55 percent in a window full of near-match paragraphs.
A worked example from Halvern Logistics
Early in an investigation into late refrigerated shipments from Ashcombe, the depot that handles only refrigerated freight, the loop retrieves a contract clause at step 2. It says refrigerated freight may only be subcontracted to a carrier with a valid cold-chain licence.
At step 3, the loop reads the register entry for the cheapest carrier, which holds no such licence.
By step 30, the window also holds a dozen passages about a handheld scanner fault that turned out to be irrelevant, and three copies of the same delivery log:

Then the loop is asked why the Ashcombe shipments cannot move to the cheapest carrier.
The clause is still in the window, word for word, but the question never mentions licences or refrigeration, so the model has to make the link on its own, as in Part 13's weak-signal case.
When we asked this question of the full window in Experiment 2, the 1.5B answered with the opening ticket's remark, read at step 1, that the dock team suspected the scanners.
Anthropic's Applied AI team makes the same argument in a September 2025 post on context engineering for agents:

Later, the post applies the point to windows of every size:

The post treats context as a finite resource with diminishing marginal returns, and expects windows of all sizes to stay subject to pollution and relevance concerns for the foreseeable future, at least where the strongest agent performance is desired.
It describes good context engineering as "finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome".

