Skip to main content
41 min read

When the Context Window Fills Up

Part 17: What a retrieval loop should clear, compact and keep outside its window as its own evidence fills it, with implementations

👉

Recap

In Part 16, we moved retrieval inside the model's own loop.

Instead of retrieving once and answering, the model searched, read what came back, and decided whether to search again, switch tools, answer or stop.

When the Model Decides What to Retrieve
Part 16: BM25, dense retrieval, and what running an agentic retrieval loop in production actually involves, with implementation.

Part 16 measured that loop on a 40-document corpus with eight information needs. A hybrid loop that kept the full question and skipped documents it had already seen found all eight, at two calls on average.

A second policy, which added distinctive terms from the documents it had just read to its next query, never found a target. With no useful stopping rule, raising its call limit from one to eight took its average retrieved tokens from 25.5 to 200.4:

Retrieved tokens keep growing while evidence stays flat, from Part 16

Part 16 also named the problem this part takes up, context bloat, where prefill grows and the relevant excerpts compete with duplicates and irrelevant text.

Its experiments ran without a model, so a target counted as found once it was retrieved. This part adds a model that reads the window and measures whether it can still use that evidence once the window fills.

A loop that searches across a long task keeps everything it reads, so each round adds to a finite window.

Every round adds to the window, and nothing leaves
💡
This part assumes you have read Parts 13 and 16. The two ideas we need from Part 12, prefix reuse and the cache memory formula, are recapped where they come up.

General context engineering, with its write, select, compress and isolate operations, is covered in the AI Agents course lesson below, so this part stays specific to retrieval.

Middleware-Driven Context Engineering in LangChain
Part 2: Dynamic prompts, memory, compression, and isolation with middleware
1

We start with Part 13's effective length and why a bigger window does not solve the problem, then take apart the window of one recorded investigation.

2

Four ways of making room follow: clearing old tool results, compacting the history into a summary, keeping the record outside the window, and giving sub-tasks windows of their own.

3

The hands-on replays the same 40-step investigation under the first three, with a 1.5B model reading every window and Claude as a second writer and reader, and prices them at production scale.

The policy that did best under a budget kept the full record and searched it when a question arrived.


Why a bigger window does not fix it

When a retrieval loop overruns its window, the obvious fix is a model with a bigger one.

That moves the overflow much later, but context bloat starts long before the window is full. Besides the growing prefill, it hurts the evidence in two ways:

On the left of the figure below, the clause the loop needs competes with passages that no longer help. On the right, every passage helps, and the model still uses the same clause less well at step 30 than at step 3:

Two ways a full window goes wrong

What Part 13 showed

On the NoLiMa benchmark, 13 models that all advertise at least 128K tokens of context degraded sharply as the context grew, on questions that share almost no words with the facts they ask about.

At 32K, 11 of them scored less than half of their short-context baseline.

Definition NoLiMa

The authors define a model's effective length as the largest tested length at which its score stays above 85 percent of its short-context score.

For every one of these models, that is far below the advertised window. Part 13 drew the idea as an illustrative curve:

Effective length, from Part 13

Why one retrieval is safe and a loop is not

Preloading, in Parts 13 to 15, ran straight into this limit, because a preloaded corpus fills the window by design.

Retrieval mostly avoids it, although on questions like these the limit sits only a few thousand tokens in. Even GPT-4o stayed above 85 percent of its baseline only up to 8K tokens, and 10 of the 13 models sat at 2K or less.

NoLiMa's effective lengths, highlighted

A single retrieval mostly gets away with a limit that low because its passages match the question, often in the question's own words.

In Part 13, a fact written in the question's own words was found at every length we tried.

A loop loses that advantage as it runs. Each round adds passages that matched an earlier query, and nothing takes them out.

The loop we replay in this part reads four passages per search. With the tool call and the model's short note on the result, a step adds a median of 567 tokens.

The window fills one round at a time

After 40 steps, the window holds 22,653 tokens, short of the 32,768 that its model, Qwen2.5-1.5B-Instruct, accepts. In the narrow sense, this loop never overflows.

At step 14, though, it passes 8,192 tokens, the longest tested length at which this model still met NoLiMa's 85 percent rule on the loop's own passages.

Smaller models reach that line sooner. In Part 13's experiment on Qwen2.5-0.5B-Instruct, a reworded fact was missed at two of three depths with only 512 tokens of filler.

Failure one: pollution

Much of what a loop retrieves stops being useful once it has been read: passages about an abandoned lead, raw tool output whose one useful number has been extracted, and near-duplicates of passages it already holds.

All of it stays in the window, and every token takes a share of the attention budget we met in Part 13.

The attention budget, from Part 13

A relevant passage gets a thinner share as the irrelevant ones pile up.

Near-matches are the worst kind of pollution in a loop, because each passage was retrieved for resembling a query and many look like the evidence the model is after. Chroma's context rot study (July 2025) found that even a single distractor lowers accuracy.

Failure two: relevance decay

Even when nothing else in the window can distract the model, it uses the relevant tokens less well as the window grows.

In "Context Length Alone Hurts LLM Performance Despite Perfect Retrieval" (October 2025), Du and colleagues measured this drop even with the irrelevant tokens masked, so that the model could attend only to the relevant ones.

What the loop's own passages do to a small model

Experiment 1 ran the 1.5B on two kinds of text.

On routine filler, length alone barely hurt. Accuracy was 99 percent at 1K tokens and 94 percent at 32K.

On the loop's own passages, full of facts of the same kind, accuracy fell from 93 percent to 66, and 133 of the 145 wrong answers gave the value of another fact of the same kind.

The wrong answer came from the next entry

The experiment does not separate length from neighbours, since on the loop's passages the two grow together.

The two failures have different fixes. Clearing junk out of the window helps with pollution, and relevance decay needs a shorter window.

Restating the evidence next to the question also helps with pollution, without removing anything.

In a pilot with the 1.5B on 150 HotpotQA questions, each needing two paragraphs joined, restating the two answering paragraphs beside the question lifted accuracy from 40 to 55 percent in a window full of near-match paragraphs.

A worked example from Halvern Logistics

Early in an investigation into late refrigerated shipments from Ashcombe, the depot that handles only refrigerated freight, the loop retrieves a contract clause at step 2. It says refrigerated freight may only be subcontracted to a carrier with a valid cold-chain licence.

At step 3, the loop reads the register entry for the cheapest carrier, which holds no such licence.

By step 30, the window also holds a dozen passages about a handheld scanner fault that turned out to be irrelevant, and three copies of the same delivery log:

The clause is still there at step 30

Then the loop is asked why the Ashcombe shipments cannot move to the cheapest carrier.

The clause is still in the window, word for word, but the question never mentions licences or refrigeration, so the model has to make the link on its own, as in Part 13's weak-signal case.

When we asked this question of the full window in Experiment 2, the 1.5B answered with the opening ticket's remark, read at step 1, that the dock team suspected the scanners.

Anthropic's Applied AI team makes the same argument in a September 2025 post on context engineering for agents:

Anthropic's post, "a finite resource"

Later, the post applies the point to windows of every size:

Anthropic's post, "context windows of all sizes"

The post treats context as a finite resource with diminishing marginal returns, and expects windows of all sizes to stay subject to pollution and relevance concerns for the foreseeable future, at least where the strongest agent performance is desired.

It describes good context engineering as "finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome".

Published on Nov 1, 2026