Skip to main content
53 min read

When the Model Decides What to Retrieve

Part 16: BM25, dense retrieval, and what running an agentic retrieval loop in production actually involves, with implementation.

👉

Fifteen parts of this course have covered how retrieval works. We chunked documents, embedded them, ranked them, compressed them, and even moved their computation before the query arrived.

One architectural choice stayed fixed through all of that work.

Retrieval happened before generation.

  • Parts 1 through 11 used the familiar RAG pipeline. A query arrived. The retriever selected a fixed number of chunks. The language model received those chunks and wrote an answer. Graph retrieval, ColBERT, ColPali, and reranking improved the selection, but they preserved this order.
  • Part 12 showed that reading retrieved tokens can cost more than finding them.
  • Parts 13 through 15 moved more work ahead of the request. We preloaded a corpus, compressed the resulting state, and studied methods that train memory into reusable artifacts.

Those systems still decide the context before the model begins answering. If retrieval misses the useful document, generation starts with the wrong evidence. The model may notice that something is missing, but the pipeline has already moved past retrieval.

Agentic retrieval changes that order.

Say the model receives search tools instead of one finished context bundle.

  • It can search for auth flow, inspect the returned files, notice that they discuss dashboards rather than redirects, and issue a better query.
  • It may switch from semantic search to an exact symbol lookup.
  • It may read one section of a file instead of loading the whole document.
  • It may also stop after the first call.
Side-by-side architecture. The left side shows a one-pass RAG pipeline with query, retriever, fixed top-k context, and generator. The right side shows the model choosing search, read, refine, switch tool, or stop after each result. Highlight that the retriever is one component inside the loop rather than the controller.

Retrieval is no longer one stage. It is a control loop inside the model's trajectory.

That change creates two consequences for engineers:

  • First, the system gains another trainable component. A model can learn when to search, what query to issue, which tool to choose, and when to stop. Reinforcement learning can reward the whole sequence when it produces a correct answer. A frozen embedding model cannot make those decisions because it only ranks documents for a query it receives.
  • Second, retrieval gains failure modes that a one-pass pipeline does not have. The model can repeat searches that return the same evidence. It can follow one weak clue until the budget expires. It can fill its context window with results that never support the answer.

This chapter develops that control loop from the ground up

We will first separate the search policy from the retriever. Then we will build lexical and dense retrieval with a small worked example.

Finally, we will measure what happens when repeated search has weak feedback and no useful stopping condition.

💡
The reader only needs the basic RAG model from Part 1. BM25, TF-IDF, latent semantic analysis, reciprocal rank fusion, and the agent loop are explained here before they are used.

From a pipeline stage to a control loop

A traditional RAG request contains one retrieval decision. The application takes the user's question, converts it into a search query, asks for the top $k$ chunks, and places those chunks in the prompt.

Suppose the user asks this question.

Why does the login service lose the destination after the consent screen?

The application may embed the full sentence and retrieve five chunks. Those five chunks are final for that request. If they describe general authentication architecture instead of redirect state, the language model cannot ask the retriever to try again unless the developer built a separate retry path.

An agent starts with a different interface.

It sees a set of operations such as text search, semantic search, file listing, and bounded reading. The model chooses one operation and supplies the query.

One trajectory may look like this:

Sequence diagram with four participants named user, model, harness, and corpus. Show the user asking once, followed by two search calls and one bounded read. Mark model decisions separately from harness execution.
  1. Search for auth flow.
  2. Read three matching excerpts.
  3. Reject two dashboard files as irrelevant.
  4. Search for consent redirect destination.
  5. Read the identity service implementation.
  6. Answer with the matching code path.

The second query exists because the first result exposed better vocabulary. The agent learned that the corpus uses destination and consent, then added those terms. A fixed pipeline cannot make this correction within the same generation.

Anthropic calls this pattern just-in-time context loading. The agent keeps lightweight references such as paths, links, and stored queries. It loads the referenced data when the current step needs them.

This does not require raw filesystem search. A tool can call BM25, a vector database, a SQL engine, or a graph index. The change is who schedules retrieval. The model chooses the next action after seeing the previous result.

What the model and harness each control

A language model cannot open a file by producing ordinary prose. The surrounding application must describe a tool, accept a structured request, run the operation, and return the result.

The model controls these choices:

  • Whether another search is needed
  • Which retrieval tool to call
  • What query to send
  • Which result deserves a deeper read
  • Whether the available evidence is sufficient

The harness controls a different set of choices:

  • Which tools the model can access
  • What arguments each tool accepts
  • How many results each call returns
  • How results enter the context window
  • Which budgets and stopping rules are enforced

Consider a literal search tool. The harness may expose it with this contract.

{
  "name": "find_text",
  "arguments": {
    "pattern": "string",
    "path": "string",
    "max_results": "integer"
  }
}

The above JSON block is the tool contract. It tells the model which tool is available and what inputs the tool accepts.

  • name identifies the operation the model can request. Here, find_text means that the tool searches for text inside files.
  • arguments defines the inputs required to run that operation. The values "string" and "integer" describe the expected data types. They are not the actual values passed to the tool.
  • In this case, the tool accepts three inputs:
    • pattern specifies the text or expression to search for.
    • path limits the search to a particular file or directory.
    • max_results limits how many matches the tool should return.

Seeing this, the model might respond with a structured call.

{
  "name": "find_text",
  "arguments": {
    "pattern": "consent.*destination|destination.*consent",
    "path": "services/identity",
    "max_results": 10
  }
}

This is an actual tool call produced by the model. Instead of answering the user directly, the model is asking the harness to execute find_text with specific values.

Sequence diagram with four participants named user, model, harness, and corpus. Show the user asking once, followed by two search calls and one bounded read. Mark model decisions separately from harness execution.
  • Here, the model asks the tool to search inside services/identity and return no more than ten matches.
  • Assuming the tool supports regular expressions, the pattern consent.*destination|destination.*consent searches for text in which consent appears before destination, or destination appears before consent.

The harness receives this structured call, checks that it matches the tool contract, runs the search, and returns the results to the model. The model can then inspect those results and decide whether it has enough evidence to answer or needs another tool call.

Why the search policy is learnable

It helps to separate a generic retriever from the search policy.

The retriever answers one narrow question → Given query $q$, which documents should rank highest?

  • BM25 answers with term statistics.
  • A dense retriever answers with vector similarity.

Both return an ordered list.

The search policy answers a wider question → Given the user's request, the evidence already seen, and the remaining budget, what should happen next?

We can write that decision as $a_t = \pi(s_t)$.

  • The state $s_t$ contains the current information gap, previous queries, returned documents, accepted evidence, and remaining budget.
  • The policy $\pi$ selects action $a_t$. And that action might issue a query, read a document, switch tools, or stop.

At the next step, the tool result changes the state. The policy makes another decision using $s_{t+1}$. The complete run becomes a trajectory.

Trajectory equation, annotated: tau is one full run, s is the state, a is the action taken, o is the tool result, and the trailing pair is the final step

Here:

  • $s_t$ is the state at step $t$.
  • The value $a_t$ is the selected action.
  • The value $o_t$ is the observation returned by the tool.
  • The final step $T$ ends with an answer or an explicit failure.
Two training paths. The first path shows query and document pairs training an embedding model that outputs a ranking. The second shows an entire search trajectory receiving a final reward that updates query generation, tool selection, evidence reading, and stopping decisions.

This representation matters because a training system can score the trajectory.

  • A correct, supported answer may receive a positive reward.
  • An unsupported answer may receive zero.
  • Extra tool calls can receive a small cost penalty
  • The training process then increases the probability of action sequences that solve the task with less wasted work.

Search-R1 paper actually demonstrates this idea.

It trains a language model to alternate reasoning with real-time search. The reward comes from final-answer correctness. The model receives no separate label telling it the perfect query at every step.

The paper masks retrieved tokens during optimization. This prevents the training algorithm from treating text produced by the search engine as if the language model generated it. Gradients update the model's own reasoning, query, and answer tokens instead.

The reported gains were 24 percent for Qwen2.5-7B and 20 percent for Qwen2.5-3B over RAG baselines under the paper's matched setting. Those results do not mean every retrieval loop should use reinforcement learning. They show that search timing and query reformulation can be optimized as model behavior.

💡
Training an embedding model improves the ranking for a supplied query. Training a search policy can change the query itself, the chosen retriever, the number of calls, and the stopping point. A production system may train either layer or both.

The policy only helps when its queries carry enough information for the available retrievers.


Why agent queries are hard

An agent rarely copies the user's full request into every search call. It searches while diagnosing the task, so each query reflects its current hypothesis.

Two training paths. The first path shows query and document pairs training an embedding model that outputs a ranking. The second shows an entire search trajectory receiving a final reward that updates query generation, tool selection, evidence reading, and stopping decisions.

Two shapes of the same question

Return to the login example we briefly discussed above. The user supplied a detailed question.

Why does the login service lose the destination after the consent screen?

The full sentence contains several useful constraints. It names a login service, a lost destination, and a consent screen. Those terms separate the problem from monitoring, permissions, session creation, and test fixtures.

The user's detailed question compressed into the agent's short probe. The full sentence is shown with its three constraints marked, naming the login service, the lost destination, and the consent screen. An arrow down to the two-word probe auth flow keeps only the broad topic, and the three dropped constraints fall away beside it, with consent marked as possibly the only term shared with the target implementation

The agent may first reduce the task to auth flow. That query names the topic but drops the failure. It does not say that the destination disappears. It also removes the consent screen, which may be the only term shared with the target implementation.

The two queries refer to the same task, but they present different retrieval problems.

Query Information preserved Information removed
Why does the login service lose the destination after the consent screen? Component, failure, stored value, execution point Little
auth flow Broad component and topic Failure, destination, consent, timing

The second query gives a retriever fewer ways to reject distractors. An authentication dashboard, migration note, runbook, and test fixture can all match auth flow. Only the implementation that stores redirect state answers the question.

The problem is not the number of words alone. ERR_CONNECTION_RESET is short but highly specific. handle error in the service after the recent change is longer but still vague. Retrieval degrades when the query omits the constraints that separate the target from nearby documents.

The CoREB benchmark tested this issue in code search. Its short search queries averaged 19 tokens, which is longer than auth flow. Every tested embedding model still fell near zero nDCG@10. The paper reports a drop of roughly two orders of magnitude from its long canonical queries.

That result should change how we interpret an agent's failed search. The embedding model may be fine. The agent may have removed the terms that made the target identifiable.

Our experiment needs a metric that penalizes buried answers.

Measuring where the answer landed

Reciprocal rank measures the position of the first correct document. Rank 1 receives full credit. Lower positions receive progressively less credit.

Reciprocal rank equation, annotated: RR is the score for one query, the numerator gives full credit at rank 1, and the denominator is the position of the first correct hit

Here, $RR$ is the reciprocal rank for one query. The denominator is the first correct document's rank. A missing correct document receives zero.

  • Rank 1 gives $1.0$.
  • Rank 2 gives $0.5$.
  • Rank 5 gives $0.2$.
  • Rank 30 gives roughly $0.033$.
A ranked list of ten documents with one marked correct at position four, alongside the resulting reciprocal rank value, illustrating that the measure depends only on where the first correct item sits

This weighting matches a top-ranked retrieval workflow. Moving the answer from rank 1 to rank 2 matters. Moving it from rank 29 to rank 30 rarely changes what the model reads.

Averaging reciprocal rank across all queries gives mean reciprocal rank, or MRR.

Plot of reciprocal rank against rank position from 1 to 30, showing a steep hyperbolic drop from 1.0 to near zero, with the region beyond rank 10 shaded as practically unread
💡
MRR only credits the first correct document. That fits our synthetic corpus because each question has one target. CoREB uses normalized discounted cumulative gain because a code query may have several useful results with different relevance grades.

Normalized discounted cumulative gain gives more credit to relevant documents near the top. It can also credit several useful results. The normalized form divides by the best possible ranking, so scores fall between zero and one.

Why under-specified probes hurt retrieval

The query auth flow states only a broad topic. It does not state what the agent needs to learn about that topic.

A corpus may contain an architecture overview, migration plan, dashboard guide, test fixture, and incident runbook for authentication. Every document matches auth flow. Only one explains how redirects survive a consent screen.

One detailed user question splitting into three possible agent probes. Show one probe preserving the failure constraint, one broad topical probe, and one exact identifier. Under each probe, show the types of documents it can distinguish.
  • BM25 rewards documents that contain auth and flow. The distractors use both words, so they receive a strong lexical score. The useful implementation may use identity service, consent, and destination instead. It can receive no lexical score at all.
  • A dense retriever faces the opposite version of the problem. It can connect auth with login and identity. That helps with vocabulary mismatch. Yet every authentication document occupies a similar semantic region. Without destination or consent, the vector does not specify which part of that region matters.

Adding more retrievers cannot restore information absent from the query. The system must add that information before retrieval or recover it through another search.

There are four common ways to recover the missing constraint.

  1. Preserve more of the user's original wording.
  2. Ask the user a clarifying question.
  3. Use the first results to discover corpus-specific vocabulary.
  4. Search another representation, such as file paths or symbols.

The third option creates the loop. It is also risky. Terms copied from a bad first result can make the second query more confidently wrong.

The hands-on experiment isolates this effect. It keeps the corpus and target fixed while changing only the query form. MRR falls by 89.2 to 93.4 percent across the tested retrievers.

A retrieval loop gives the agent another chance to add constraints. It helps only when the new query contains information missing from the first.


The two retrieval families

The experiment compares lexical retrieval, dense retrieval, and their fused ranking.

Lexical retrieval and BM25

Lexical retrieval compares the query with the words stored in each document. It does not need to infer that login and authentication are related. A document receives credit when the query terms appear in its text.

Consider three short documents.

Document Text
D1 auth service preserves redirect after consent
D2 auth dashboard tracks login failures
D3 billing service retries failed charges

Now search for auth redirect.

  • Document D1 contains both query terms.
  • Document D2 contains only auth.
  • Document D3 contains neither.

A plain word-overlap count would rank D1 first, D2 second, and D3 last.

But that baseline misses an important clue. redirect appears in only one document, while auth appears in two. Matching redirect should therefore carry more information.

Inverse document frequency, or IDF, gives rare terms more weight, and one common smoothed form is shown below.

Inverse document frequency equation, annotated: N is the total documents, df of t is the documents containing t, and the trailing plus one keeps the weight positive
  • The value $N$ is the number of documents.
  • The value $\text{df}(t)$ counts how many documents contain term $t$. Adding one prevents division by zero and keeps the value defined for small corpora.

For our three documents:

  • auth has a document frequency of two. Its IDF is about $1.29$.
  • The term redirect has a document frequency of one. Its IDF is about $1.69$.

Document D1 receives both weights. Document D2 receives only the smaller auth weight. The rare term makes D1 easier to identify.

Term frequency, or TF, supplies the other half of TF-IDF. It measures how often a term appears inside one document. A basic TF-IDF weight multiplies the within-document count by the corpus-level rarity.

TF-IDF equation, annotated: TF is the count inside this document and IDF is the rarity across the corpus

The value $\text{TF}(t,D)$ counts term $t$ inside document $D$. The IDF term stays the same across documents. A term receives a large weight when it appears often in one document but rarely across the corpus.

TF-IDF represents every document as a sparse vector. Each column corresponds to one vocabulary term. Most entries are zero because one document uses only a small part of the corpus vocabulary.

For the three-document example, a small part of that matrix looks like this:

Worked TF-IDF example using the three documents from the table. Show document frequency, the resulting IDF for auth and redirect, the sparse matrix, and why D1 ranks above D2 for the query auth redirect.
Document auth redirect billing failures
D1 1.29 1.69 0 0
D2 1.29 0 0 1.69
D3 0 0 1.69 0

The real matrix contains one column per vocabulary term. Search compares the query vector with these document vectors. A shared rare term creates a strong match.

TF-IDF has two problems:

  • TF-IDF still grows linearly when a word repeats.
  • A document containing redirect twenty times can receive twenty times the raw term-frequency contribution. Long documents also have more chances to contain every query word.

BM25 modifies the weighting to handle those two problems. It combines term rarity, saturating term frequency, and document-length normalization.

  • Term frequency counts how often a query term appears. Repeated occurrences increase the score, but each additional occurrence contributes less. This prevents keyword stuffing from increasing the score without limit.
  • Inverse document frequency, or IDF, weights terms by rarity. A term found in one document identifies it better than a term found everywhere.
  • Length normalization corrects the advantage of long documents. Longer text has more opportunities to contain every query word.

BM25 combines these quantities as follows.

BM25 scoring equation, annotated: the summation covers every query term, IDF is term rarity, f of t and D is the term count in the document, k1 controls saturation, and the bracketed group applies length correction
  • The sum iterates every query term $t$.
  • The function $f(t,D)$ counts that term inside document $D$.
  • The value $|D|$ is the document length.
  • The value $\text{avgdl}$ is the average number of indexed tokens per document across the corpus. BM25 compares the length of the current document, $\lvert d\rvert$, against this average.
  • The constants $k_1$ and $b$ control term-frequency saturation and document-length correction.

The fraction saturates term frequency. Both numerator and denominator grow with $f(t,D)$. Its contribution therefore approaches a ceiling instead of growing forever.

Diagram showing the three ingredients of BM25 combining into one score: a saturating curve for term frequency (diminishing returns per extra mention), an inverse-document-frequency scale weighting rare terms higher than common ones, and a length-normalisation adjustment shrinking scores for longer-than-average documents, all feeding into the final BM25 score for a document

The bracket applies length normalization. An average-length document makes $|D|/\text{avgdl}$ equal one. Longer documents increase the denominator and receive a smaller score.

Setting $b=0$ disables this correction.

The IDF multiplier (in green) then gives rare matches more weight:

BM25 scoring equation, annotated: the summation covers every query term, IDF is term rarity, f of t and D is the term count in the document, k1 controls saturation, and the bracketed group applies length correction

Return to document D1 above.

  • One occurrence of redirect is already useful because the term is rare.
  • A second occurrence increases the score, but not by the same amount.
  • The fraction approaches a ceiling as the frequency grows.

Now imagine D1 is a 5,000-word architecture document while D2 is a 20-word incident note.

BM25 compares each length with the corpus average. The long document receives a correction because accidental matches become more likely as text grows.

The parameters $k_1$ and $b$ control these behaviors. A larger $k_1$ allows repeated terms to keep adding value for longer. The parameter $b$ controls length normalization. Setting $b$ to zero removes the length correction. A common starting point is $k_1=1.2$ to $2.0$ and $b=0.75$, but the best values depend on the corpus.

This image summarizes the full formula again:

BM25 works well for error codes, identifiers, filenames, product names, and exact phrases. It fails when the target expresses the same concept with different words. A query for billing retry cannot match PaymentAttemptBackoff if neither query term appears in the document.

💡
Grep and BM25 are both lexical, but they are not the same tool. Grep usually applies a literal or regular-expression condition. BM25 ranks all documents using term statistics. An agent may use grep for a known symbol and BM25 for a broader keyword search.

Dense retrieval and the latent space

Dense retrieval maps queries and documents into vectors. Related text should receive nearby vectors, even when its words differ.

Production systems usually use trained transformer encoders. Parts 1 and 5 explained how embedding models learn these representations.

Our local experiment uses latent semantic analysis (LSA), so it runs without a model download.

Two-panel diagram. Left shows a sparse TF-IDF matrix with documents as rows and a very wide vocabulary as columns, mostly zeros. Right shows the same documents after projection into a small number of dense dimensions, with topically related documents clustered together

LSA first builds a TF-IDF matrix. Each row represents one document, and each column represents one term. Singular value decomposition (SVD) then projects this sparse matrix into 24 dense dimensions.

💡
For more context, SVD is commonly used for dimensionality reduction because it preserves the most important patterns in a matrix while representing it with fewer dimensions.

In this case, SVD factorizes the TF-IDF matrix as ($X = U \Sigma V^ \top$). The columns of (V) define directions that capture recurring patterns of terms across documents, while the singular values in ($\Sigma$) rank those directions by importance.

LSA keeps the top 24 directions and represents each document using its coordinates along them. This replaces thousands of sparse term features with 24 dense values.

The projection compresses vocabulary columns that behave similarly across documents. Suppose many documents containing login also contain identity, session, and consent. Their columns develop related patterns. LSA can represent that shared pattern with one latent dimension instead of four independent word columns.

This is why the TF-IDF example comes first. LSA does not ignore lexical statistics. It starts with them, then compresses their co-occurrence structure.

Imagine a corpus containing these phrases across many documents:

  • login session expired
  • identity session created
  • authentication session cookie
  • user consent during login

The words login, identity, and authentication do not appear together in every document. They do appear around overlapping terms such as session, user, and consent.

LSA can use those repeated patterns to place their respective documents closer after projection.

The word "latent" refers to these hidden directions. A dimension does not have to correspond to one named topic. It is a weighted combination of many vocabulary terms that explains variation across the matrix.

For a query, LSA performs the same TF-IDF transformation and projection. Cosine similarity then compares the query direction with every document direction. Documents pointing in a similar direction rank higher.

LSA has clear limits. It needs enough co-occurrence evidence in the corpus. It also represents a word with the same corpus-level pattern in every context. Transformer embeddings handle context and language variation more directly.

💡
LSA is not a substitute for a trained embedding model. We use it to isolate the experiment without network access or a GPU. The measured scores apply only to this implementation and corpus.

Combining them without calibration

Lexical and dense retrieval fail on different queries.

  • Lexical search favors exact wording.
  • Dense search can recover paraphrases but may rank nearby topics together.

As you may have guessed, their raw scores cannot be added directly since:

  • BM25 scores are unbounded positive values.
  • Cosine similarity usually lies between minus one and one.
  • A value of 0.8 therefore means something different in each system.

Reciprocal rank fusion, or RRF, combines their ranked positions instead.

Reciprocal rank fusion equation, annotated: RRF of d is the fused score, the summation covers each ranked list, k is the damping constant, and rank r of d is the position in list r

Here:

  • $d$ is one document.
  • $R$ contains the ranked lists. In our example, one list orders documents by their BM25 scores, while the other orders them by cosine similarity.
  • $r$ refers to one particular ranked list in $R$.
  • The term $\text{rank}_r(d)$ is the document's position in list $r$.
  • The constant $k$ reduces the difference between adjacent ranks.

RRF ignores the original BM25 and cosine-similarity scores. It only uses the position of each document in the two lists.

Each list contributes $1/(k+\text{rank})$ to the document. High positions contribute more. Documents ranked well by both retrievers accumulate the largest total.

Without $k$, rank 1 contributes $1.0$ and rank 2 contributes $0.5$. With $k=60$, they contribute roughly $0.0164$ and $0.0161$. Agreement across retrievers then matters more than one first-place result.

Let's consider an example.

Suppose BM25 ranks documents A, B, and C. Dense retrieval ranks B, D, and A.

Document B receives contributions from ranks two and one. Document A receives contributions from ranks one and three. Document C appears only in BM25, while D appears only in dense retrieval.

Document BM25 rank Dense rank RRF contribution with $k=60$
A 1 3 $1/61 + 1/63$
B 2 1 $1/62 + 1/61$
C 3 Not returned $1/63$
D Not returned 2 $1/62$

Document B wins because both retrievers place it near the top. The calculation never compares a BM25 score with cosine similarity.

The reciprocal rank fusion worked example. Two ranked lists, one from BM25 holding A, B, C and one from dense retrieval holding B, D, A, have each position turned into one over sixty plus the rank. The contributions are summed per document, and B finishes first because it placed near the top in both lists while A, C and D fall behind

RRF needs no score calibration. It can combine any retrievers that return ordered document lists.

The experiment now has three rankings to compare against under-specified probes.


Evidence that supports model-directed retrieval

The strongest case for model-directed retrieval does not come from a single retriever but rather from systems where the model can run several cheap searches and adapt after each result.

Keyword tools can recover much of a RAG baseline

An Amazon study compared conventional RAG with a tool-using agent. The agent could search documents with keyword tools but had no standing vector database.

The keyword system reached more than 90 percent of the RAG system across the reported metrics. The faithfulness result was 94.5 percent of the RAG score. Context recall reached 88.0 percent, while answer correctness reached 91.5 percent of the RAG score.

To clarify, these are relative attainments, not raw accuracy values. A 94.5 percent attainment means the keyword agent retained most of the reference RAG score on that metric. It does not mean 94.5 percent of all answers were faithful.

Bar comparison of the keyword agent against the RAG baseline on three metrics, with faithfulness at 94.5 percent, answer correctness at 91.5 percent, and context recall at 88.0 percent of the baseline score. The baseline is drawn as a full-height reference bar and the axis is labelled as a share of the baseline rather than absolute accuracy

The mechanism is simple. A one-pass keyword query may miss. A model can read the misses, extract a better name or phrase, and search again. Several adaptive lexical searches can recover evidence that one fixed lexical query would miss.

Frequently changing corpora benefit from this design. A filesystem search sees the current text. A vector index only sees content processed by its latest ingestion job. Removing the standing index also removes chunk embedding, index storage, and synchronization work.

That saving moves the actual work to request time. Every agent search consumes latency and model tokens. A stable corpus with high query volume may still benefit from paying the indexing cost once.

💡
The Amazon result supports keyword tools as a viable retrieval path. It does not show that semantic retrieval is unnecessary for every corpus or query class.

Policy training improves the loop itself

The Search-R1 paper is another solution where the retriever, training data, and base model were held fixed. Reinforcement learning changed how the model used search.

The Qwen2.5-7B system improved by 24 percent over the matched RAG baselines across seven question-answering datasets. The Qwen2.5-3B system improved by 20 percent. The model learned to issue multiple queries during reasoning and use the returned evidence before answering.

Putting retrieval inside the trajectory makes those decisions trainable. The ranking function may remain unchanged while the policy improves search timing, query wording, and evidence use.

Context freshness due to tools

Anthropic's just-in-time context pattern keeps identifiers in context and loads data during the task. A path points to the current file. A stored query can read the current database state. A web link can fetch the current page.

Cost placement comparison. Show an indexed path paying chunking and embedding costs during ingestion, then cheap retrieval per request. Show a raw-text agent paying little ingestion cost but repeated search and model costs during each request.

An index can also be exposed through the same tool interface. The agent may use raw text for fresh local files and semantic search for a stable knowledge base. Model-directed retrieval does not force one storage design.


Evidence against keyword-only retrieval

The tempting conclusion is that agents made vector search obsolete. But if you holistically consider all the available evidence, there are problems with keyword-only retrieval.

Dense retrieval cannot recover details missing from the query

Semantic search is often presented as the solution to vocabulary mismatch.

A query may use billing retry while the code uses PaymentAttemptBackoff. An embedding model can place both near each other even though they share no words.

That ability has a limit. The query still needs to describe what the search should distinguish.

The CoREB benchmark shared some details on this.

CoREB evaluates code retrieval in three directions:

A text description can retrieve code, code can retrieve its description, and one code sample can retrieve related code. The result relevant here comes from its text-to-code task.

That task contains two different query formats.

  • The Canonical queries are full problem descriptions.
  • The Search queries are shorter, developer-style keyword descriptions averaging about 19 tokens.

Both search the same corpus of 1,744 code items, so corpus size and candidate difficulty remain fixed.

Before looking at the scores, we need to understand the metric.

nDCG@10 stands for normalized discounted cumulative gain at 10.

It evaluates the first ten retrieved items. Relevant results earn gain, but results near the top earn more than results near position ten. CoREB also uses graded relevance, so a correct solution receives more credit than a related but failed solution.

The score is normalized against the best possible ordering for that query.

  • A score of 1 means the first ten results are ordered ideally.
  • A lower score means useful results appear late or below weaker results.
  • A score near 0 means the ranking captures almost none of the useful signal available in an ideal top ten.

Suppose the correct implementation appears at rank one. That ranking receives strong credit. If four failed solutions appear first and the correct implementation sits at rank eight, the score falls. If the correct implementation never enters the first ten, the ranking receives no gain from it.

CoREB's long Canonical queries produced nDCG@10 scores between 0.450 and 0.582 across the eleven embedding models. The shorter Search queries produced scores between 0.000 and 0.015. Even the best Search result reached only 0.015.

This is a drop of roughly two orders of magnitude. Larger embedding models did not avoid it. Code-specialized models did not avoid it either. The paper also reports that reranking did not close this gap.

The result does not show that semantic retrieval is ineffective. The same models performed far better when the query preserved the full problem description. It shows that semantic similarity cannot reconstruct requirements that never entered the query.

Consider auth flow again. An embedding model can retrieve authentication code even if the repository uses words such as identity, login, or session. But the query does not say that the destination disappears after consent.

The model therefore has no reason to prefer redirect-state code over authentication dashboards, session middleware, or login tests. All of them are semantically related to authentication.

BM25 and dense retrieval fail differently here. BM25 may miss the target because the query words never appear in it. Dense retrieval may find the correct topic but return many neighboring files because the query omits the property that separates the answer.

The engineering conclusion is specific. Do not reduce a detailed information need to a broad topic label before semantic search. Pass the full question, or build a query that preserves the component, behavior, failure, and relevant constraint. If an agent generates auth flow, the search policy has already discarded most of the information the retriever needed.

Why semantic search still belongs in the system

The CoREB result tells us not to expect an embedding model to recover constraints that the agent removed from the query. It does not remove the problem embeddings were built to solve, i.e., the words in the request and the words in the corpus often differ.

Suppose a developer asks:

Where do we retry a failed card charge?

And the relevant implementation in code might look like this:

class PaymentAttemptBackoff:
    def schedule_next_attempt(self, attempt):
        delay = min(2 ** attempt.count, 3600)
        enqueue(attempt.invoice_id, delay=delay)

The query contains retry, failed, card, and charge. The code contains none of those words.

A literal search for the full question may return documentation about payment failures, UI copy for declined cards, or nothing at all. BM25 has no direct evidence connecting retry to schedule_next_attempt or card charge to PaymentAttemptBackoff.

An agent can sometimes bridge the gap through repeated lexical searches. It might search for payment, discover PaymentAttempt, inspect nearby symbols, and then search for schedule_next_attempt.

That route works only if an early result exposes the repository's vocabulary. If the first few terms miss, the agent has nothing useful from which to reformulate the next query.

Dense retrieval offers a different first move. It embeds the natural-language request and the code or its surrounding text, then looks for items that express a similar function. Names such as backoff, next_attempt, invoice_id, and enqueue jointly describe retry behavior even though no single token matches the developer's wording.

This is the setting Cursor tested in its semantic search study. Its offline Cursor Context Benchmark compared an agent with semantic search available against an agent without it.

The agents still had traditional code-search tools, so the experiment was not semantic search versus grep in isolation. It asked whether semantic retrieval adds value when the agent can already search lexically.

Across the tested model configurations, adding semantic search increased answer accuracy by 12.5 percent on average. The improvement ranged from 6.5 to 23.5 percent depending on the model.

Cursor also ran an online A/B test because an offline retrieval task does not tell us whether developers keep the generated code. Across all requests, semantic search increased retained code by 0.3 percent. On codebases with at least 1,000 files, the increase was 2.6 percent. When semantic search was disabled, dissatisfied follow-up requests increased by 2.2 percent.

The smaller overall online gain has a straightforward explanation. Many editor requests do not need repository search. Renaming a local variable or completing the next line will not benefit from an index.

The effect becomes larger on repositories where the answer is harder to locate from the open file and its immediate neighbors.

These are Cursor's internal results, not an independent public benchmark, so they should not be treated as a universal estimate.

They still test a more useful question than "Does vector search beat grep?" The measured improvement came from giving an existing search agent another retrieval path.

Cursor's implementation also shows how semantic retrieval changes inside an agentic system. The company trains its embedding model using traces of agent sessions. An LLM reviews the files and searches that appeared later in a successful trajectory and estimates which items would have been useful earlier. Those rankings become training signal for the embedding model.

The retriever is no longer optimized only for generic text similarity. It is trained toward evidence that helps an agent complete a software task. That connects the semantic index to the control loop instead of leaving it as a frozen component in front of the model.

Published on Oct 25, 2026