When the Model Decides What to Retrieve
Part 16: BM25, dense retrieval, and what running an agentic retrieval loop in production actually involves, with implementation.
Fifteen parts of this course have covered how retrieval works. We chunked documents, embedded them, ranked them, compressed them, and even moved their computation before the query arrived.
One architectural choice stayed fixed through all of that work.
Retrieval happened before generation.
- Parts 1 through 11 used the familiar RAG pipeline. A query arrived. The retriever selected a fixed number of chunks. The language model received those chunks and wrote an answer. Graph retrieval, ColBERT, ColPali, and reranking improved the selection, but they preserved this order.
- Part 12 showed that reading retrieved tokens can cost more than finding them.
- Parts 13 through 15 moved more work ahead of the request. We preloaded a corpus, compressed the resulting state, and studied methods that train memory into reusable artifacts.
Those systems still decide the context before the model begins answering. If retrieval misses the useful document, generation starts with the wrong evidence. The model may notice that something is missing, but the pipeline has already moved past retrieval.
Agentic retrieval changes that order.
Say the model receives search tools instead of one finished context bundle.
- It can search for
auth flow, inspect the returned files, notice that they discuss dashboards rather than redirects, and issue a better query. - It may switch from semantic search to an exact symbol lookup.
- It may read one section of a file instead of loading the whole document.
- It may also stop after the first call.

Retrieval is no longer one stage. It is a control loop inside the model's trajectory.
That change creates two consequences for engineers:

- First, the system gains another trainable component. A model can learn when to search, what query to issue, which tool to choose, and when to stop. Reinforcement learning can reward the whole sequence when it produces a correct answer. A frozen embedding model cannot make those decisions because it only ranks documents for a query it receives.
- Second, retrieval gains failure modes that a one-pass pipeline does not have. The model can repeat searches that return the same evidence. It can follow one weak clue until the budget expires. It can fill its context window with results that never support the answer.
This chapter develops that control loop from the ground up
We will first separate the search policy from the retriever. Then we will build lexical and dense retrieval with a small worked example.
Finally, we will measure what happens when repeated search has weak feedback and no useful stopping condition.
From a pipeline stage to a control loop
A traditional RAG request contains one retrieval decision. The application takes the user's question, converts it into a search query, asks for the top $k$ chunks, and places those chunks in the prompt.

Suppose the user asks this question.
Why does the login service lose the destination after the consent screen?
The application may embed the full sentence and retrieve five chunks. Those five chunks are final for that request. If they describe general authentication architecture instead of redirect state, the language model cannot ask the retriever to try again unless the developer built a separate retry path.
An agent starts with a different interface.
It sees a set of operations such as text search, semantic search, file listing, and bounded reading. The model chooses one operation and supplies the query.
One trajectory may look like this:

- Search for
auth flow. - Read three matching excerpts.
- Reject two dashboard files as irrelevant.
- Search for
consent redirect destination. - Read the identity service implementation.
- Answer with the matching code path.
The second query exists because the first result exposed better vocabulary. The agent learned that the corpus uses destination and consent, then added those terms. A fixed pipeline cannot make this correction within the same generation.
Anthropic calls this pattern just-in-time context loading. The agent keeps lightweight references such as paths, links, and stored queries. It loads the referenced data when the current step needs them.

This does not require raw filesystem search. A tool can call BM25, a vector database, a SQL engine, or a graph index. The change is who schedules retrieval. The model chooses the next action after seeing the previous result.
What the model and harness each control
A language model cannot open a file by producing ordinary prose. The surrounding application must describe a tool, accept a structured request, run the operation, and return the result.
The model controls these choices:
- Whether another search is needed
- Which retrieval tool to call
- What query to send
- Which result deserves a deeper read
- Whether the available evidence is sufficient

The harness controls a different set of choices:
- Which tools the model can access
- What arguments each tool accepts
- How many results each call returns
- How results enter the context window
- Which budgets and stopping rules are enforced
Consider a literal search tool. The harness may expose it with this contract.
{
"name": "find_text",
"arguments": {
"pattern": "string",
"path": "string",
"max_results": "integer"
}
}
The above JSON block is the tool contract. It tells the model which tool is available and what inputs the tool accepts.
nameidentifies the operation the model can request. Here,find_textmeans that the tool searches for text inside files.argumentsdefines the inputs required to run that operation. The values"string"and"integer"describe the expected data types. They are not the actual values passed to the tool.- In this case, the tool accepts three inputs:
patternspecifies the text or expression to search for.pathlimits the search to a particular file or directory.max_resultslimits how many matches the tool should return.
Seeing this, the model might respond with a structured call.
{
"name": "find_text",
"arguments": {
"pattern": "consent.*destination|destination.*consent",
"path": "services/identity",
"max_results": 10
}
}
This is an actual tool call produced by the model. Instead of answering the user directly, the model is asking the harness to execute find_text with specific values.

- Here, the model asks the tool to search inside
services/identityand return no more than ten matches. - Assuming the tool supports regular expressions, the pattern
consent.*destination|destination.*consentsearches for text in whichconsentappears beforedestination, ordestinationappears beforeconsent.
The harness receives this structured call, checks that it matches the tool contract, runs the search, and returns the results to the model. The model can then inspect those results and decide whether it has enough evidence to answer or needs another tool call.
Why the search policy is learnable
It helps to separate a generic retriever from the search policy.
The retriever answers one narrow question → Given query $q$, which documents should rank highest?

- BM25 answers with term statistics.
- A dense retriever answers with vector similarity.
Both return an ordered list.
The search policy answers a wider question → Given the user's request, the evidence already seen, and the remaining budget, what should happen next?
We can write that decision as $a_t = \pi(s_t)$.
- The state $s_t$ contains the current information gap, previous queries, returned documents, accepted evidence, and remaining budget.
- The policy $\pi$ selects action $a_t$. And that action might issue a query, read a document, switch tools, or stop.
At the next step, the tool result changes the state. The policy makes another decision using $s_{t+1}$. The complete run becomes a trajectory.

Here:
- $s_t$ is the state at step $t$.
- The value $a_t$ is the selected action.
- The value $o_t$ is the observation returned by the tool.
- The final step $T$ ends with an answer or an explicit failure.

This representation matters because a training system can score the trajectory.
- A correct, supported answer may receive a positive reward.
- An unsupported answer may receive zero.
- Extra tool calls can receive a small cost penalty
- The training process then increases the probability of action sequences that solve the task with less wasted work.
Search-R1 paper actually demonstrates this idea.

It trains a language model to alternate reasoning with real-time search. The reward comes from final-answer correctness. The model receives no separate label telling it the perfect query at every step.
The paper masks retrieved tokens during optimization. This prevents the training algorithm from treating text produced by the search engine as if the language model generated it. Gradients update the model's own reasoning, query, and answer tokens instead.
The reported gains were 24 percent for Qwen2.5-7B and 20 percent for Qwen2.5-3B over RAG baselines under the paper's matched setting. Those results do not mean every retrieval loop should use reinforcement learning. They show that search timing and query reformulation can be optimized as model behavior.
The policy only helps when its queries carry enough information for the available retrievers.
Why agent queries are hard
An agent rarely copies the user's full request into every search call. It searches while diagnosing the task, so each query reflects its current hypothesis.

Two shapes of the same question
Return to the login example we briefly discussed above. The user supplied a detailed question.
Why does the login service lose the destination after the consent screen?
The full sentence contains several useful constraints. It names a login service, a lost destination, and a consent screen. Those terms separate the problem from monitoring, permissions, session creation, and test fixtures.

The agent may first reduce the task to auth flow. That query names the topic but drops the failure. It does not say that the destination disappears. It also removes the consent screen, which may be the only term shared with the target implementation.
The two queries refer to the same task, but they present different retrieval problems.
| Query | Information preserved | Information removed |
|---|---|---|
Why does the login service lose the destination after the consent screen?
|
Component, failure, stored value, execution point | Little |
auth flow |
Broad component and topic | Failure, destination, consent, timing |
The second query gives a retriever fewer ways to reject distractors. An authentication dashboard, migration note, runbook, and test fixture can all match auth flow. Only the implementation that stores redirect state answers the question.
The problem is not the number of words alone. ERR_CONNECTION_RESET is short but highly specific. handle error in the service after the recent change is longer but still vague. Retrieval degrades when the query omits the constraints that separate the target from nearby documents.

The CoREB benchmark tested this issue in code search. Its short search queries averaged 19 tokens, which is longer than auth flow. Every tested embedding model still fell near zero nDCG@10. The paper reports a drop of roughly two orders of magnitude from its long canonical queries.
That result should change how we interpret an agent's failed search. The embedding model may be fine. The agent may have removed the terms that made the target identifiable.
Our experiment needs a metric that penalizes buried answers.
Measuring where the answer landed
Reciprocal rank measures the position of the first correct document. Rank 1 receives full credit. Lower positions receive progressively less credit.

Here, $RR$ is the reciprocal rank for one query. The denominator is the first correct document's rank. A missing correct document receives zero.
- Rank 1 gives $1.0$.
- Rank 2 gives $0.5$.
- Rank 5 gives $0.2$.
- Rank 30 gives roughly $0.033$.

This weighting matches a top-ranked retrieval workflow. Moving the answer from rank 1 to rank 2 matters. Moving it from rank 29 to rank 30 rarely changes what the model reads.
Averaging reciprocal rank across all queries gives mean reciprocal rank, or MRR.

Normalized discounted cumulative gain gives more credit to relevant documents near the top. It can also credit several useful results. The normalized form divides by the best possible ranking, so scores fall between zero and one.
Why under-specified probes hurt retrieval
The query auth flow states only a broad topic. It does not state what the agent needs to learn about that topic.
A corpus may contain an architecture overview, migration plan, dashboard guide, test fixture, and incident runbook for authentication. Every document matches auth flow. Only one explains how redirects survive a consent screen.

- BM25 rewards documents that contain
authandflow. The distractors use both words, so they receive a strong lexical score. The useful implementation may useidentity service,consent, anddestinationinstead. It can receive no lexical score at all. - A dense retriever faces the opposite version of the problem. It can connect
authwithloginandidentity. That helps with vocabulary mismatch. Yet every authentication document occupies a similar semantic region. Withoutdestinationorconsent, the vector does not specify which part of that region matters.
Adding more retrievers cannot restore information absent from the query. The system must add that information before retrieval or recover it through another search.
There are four common ways to recover the missing constraint.
- Preserve more of the user's original wording.
- Ask the user a clarifying question.
- Use the first results to discover corpus-specific vocabulary.
- Search another representation, such as file paths or symbols.
The third option creates the loop. It is also risky. Terms copied from a bad first result can make the second query more confidently wrong.
The hands-on experiment isolates this effect. It keeps the corpus and target fixed while changing only the query form. MRR falls by 89.2 to 93.4 percent across the tested retrievers.
A retrieval loop gives the agent another chance to add constraints. It helps only when the new query contains information missing from the first.
The two retrieval families
The experiment compares lexical retrieval, dense retrieval, and their fused ranking.
Lexical retrieval and BM25
Lexical retrieval compares the query with the words stored in each document. It does not need to infer that login and authentication are related. A document receives credit when the query terms appear in its text.
Consider three short documents.
| Document | Text |
|---|---|
| D1 | auth service preserves redirect after consent |
| D2 | auth dashboard tracks login failures |
| D3 | billing service retries failed charges |
Now search for auth redirect.
- Document D1 contains both query terms.
- Document D2 contains only
auth. - Document D3 contains neither.
A plain word-overlap count would rank D1 first, D2 second, and D3 last.
But that baseline misses an important clue. redirect appears in only one document, while auth appears in two. Matching redirect should therefore carry more information.
Inverse document frequency, or IDF, gives rare terms more weight, and one common smoothed form is shown below.

- The value $N$ is the number of documents.
- The value $\text{df}(t)$ counts how many documents contain term $t$. Adding one prevents division by zero and keeps the value defined for small corpora.
For our three documents:
authhas a document frequency of two. Its IDF is about $1.29$.- The term
redirecthas a document frequency of one. Its IDF is about $1.69$.
Document D1 receives both weights. Document D2 receives only the smaller auth weight. The rare term makes D1 easier to identify.
Term frequency, or TF, supplies the other half of TF-IDF. It measures how often a term appears inside one document. A basic TF-IDF weight multiplies the within-document count by the corpus-level rarity.

The value $\text{TF}(t,D)$ counts term $t$ inside document $D$. The IDF term stays the same across documents. A term receives a large weight when it appears often in one document but rarely across the corpus.
TF-IDF represents every document as a sparse vector. Each column corresponds to one vocabulary term. Most entries are zero because one document uses only a small part of the corpus vocabulary.
For the three-document example, a small part of that matrix looks like this:

| Document | auth |
redirect |
billing |
failures |
|---|---|---|---|---|
| D1 | 1.29 | 1.69 | 0 | 0 |
| D2 | 1.29 | 0 | 0 | 1.69 |
| D3 | 0 | 0 | 1.69 | 0 |
The real matrix contains one column per vocabulary term. Search compares the query vector with these document vectors. A shared rare term creates a strong match.
TF-IDF has two problems:
- TF-IDF still grows linearly when a word repeats.
- A document containing
redirecttwenty times can receive twenty times the raw term-frequency contribution. Long documents also have more chances to contain every query word.
BM25 modifies the weighting to handle those two problems. It combines term rarity, saturating term frequency, and document-length normalization.
- Term frequency counts how often a query term appears. Repeated occurrences increase the score, but each additional occurrence contributes less. This prevents keyword stuffing from increasing the score without limit.
- Inverse document frequency, or IDF, weights terms by rarity. A term found in one document identifies it better than a term found everywhere.
- Length normalization corrects the advantage of long documents. Longer text has more opportunities to contain every query word.
BM25 combines these quantities as follows.

- The sum iterates every query term $t$.
- The function $f(t,D)$ counts that term inside document $D$.
- The value $|D|$ is the document length.
- The value $\text{avgdl}$ is the average number of indexed tokens per document across the corpus. BM25 compares the length of the current document, $\lvert d\rvert$, against this average.
- The constants $k_1$ and $b$ control term-frequency saturation and document-length correction.
The fraction saturates term frequency. Both numerator and denominator grow with $f(t,D)$. Its contribution therefore approaches a ceiling instead of growing forever.

The bracket applies length normalization. An average-length document makes $|D|/\text{avgdl}$ equal one. Longer documents increase the denominator and receive a smaller score.
Setting $b=0$ disables this correction.
The IDF multiplier (in green) then gives rare matches more weight:

Return to document D1 above.
- One occurrence of
redirectis already useful because the term is rare. - A second occurrence increases the score, but not by the same amount.
- The fraction approaches a ceiling as the frequency grows.
Now imagine D1 is a 5,000-word architecture document while D2 is a 20-word incident note.
BM25 compares each length with the corpus average. The long document receives a correction because accidental matches become more likely as text grows.
The parameters $k_1$ and $b$ control these behaviors. A larger $k_1$ allows repeated terms to keep adding value for longer. The parameter $b$ controls length normalization. Setting $b$ to zero removes the length correction. A common starting point is $k_1=1.2$ to $2.0$ and $b=0.75$, but the best values depend on the corpus.
This image summarizes the full formula again:

BM25 works well for error codes, identifiers, filenames, product names, and exact phrases. It fails when the target expresses the same concept with different words. A query for billing retry cannot match PaymentAttemptBackoff if neither query term appears in the document.
Dense retrieval and the latent space
Dense retrieval maps queries and documents into vectors. Related text should receive nearby vectors, even when its words differ.

Production systems usually use trained transformer encoders. Parts 1 and 5 explained how embedding models learn these representations.
Our local experiment uses latent semantic analysis (LSA), so it runs without a model download.

LSA first builds a TF-IDF matrix. Each row represents one document, and each column represents one term. Singular value decomposition (SVD) then projects this sparse matrix into 24 dense dimensions.
In this case, SVD factorizes the TF-IDF matrix as ($X = U \Sigma V^ \top$). The columns of (V) define directions that capture recurring patterns of terms across documents, while the singular values in ($\Sigma$) rank those directions by importance.
LSA keeps the top 24 directions and represents each document using its coordinates along them. This replaces thousands of sparse term features with 24 dense values.
The projection compresses vocabulary columns that behave similarly across documents. Suppose many documents containing login also contain identity, session, and consent. Their columns develop related patterns. LSA can represent that shared pattern with one latent dimension instead of four independent word columns.

This is why the TF-IDF example comes first. LSA does not ignore lexical statistics. It starts with them, then compresses their co-occurrence structure.
Imagine a corpus containing these phrases across many documents:
login session expiredidentity session createdauthentication session cookieuser consent during login
The words login, identity, and authentication do not appear together in every document. They do appear around overlapping terms such as session, user, and consent.

LSA can use those repeated patterns to place their respective documents closer after projection.
The word "latent" refers to these hidden directions. A dimension does not have to correspond to one named topic. It is a weighted combination of many vocabulary terms that explains variation across the matrix.
For a query, LSA performs the same TF-IDF transformation and projection. Cosine similarity then compares the query direction with every document direction. Documents pointing in a similar direction rank higher.
LSA has clear limits. It needs enough co-occurrence evidence in the corpus. It also represents a word with the same corpus-level pattern in every context. Transformer embeddings handle context and language variation more directly.
Combining them without calibration
Lexical and dense retrieval fail on different queries.
- Lexical search favors exact wording.
- Dense search can recover paraphrases but may rank nearby topics together.
As you may have guessed, their raw scores cannot be added directly since:

- BM25 scores are unbounded positive values.
- Cosine similarity usually lies between minus one and one.
- A value of
0.8therefore means something different in each system.
Reciprocal rank fusion, or RRF, combines their ranked positions instead.

Here:
- $d$ is one document.
- $R$ contains the ranked lists. In our example, one list orders documents by their BM25 scores, while the other orders them by cosine similarity.
- $r$ refers to one particular ranked list in $R$.
- The term $\text{rank}_r(d)$ is the document's position in list $r$.
- The constant $k$ reduces the difference between adjacent ranks.
RRF ignores the original BM25 and cosine-similarity scores. It only uses the position of each document in the two lists.
Each list contributes $1/(k+\text{rank})$ to the document. High positions contribute more. Documents ranked well by both retrievers accumulate the largest total.
Without $k$, rank 1 contributes $1.0$ and rank 2 contributes $0.5$. With $k=60$, they contribute roughly $0.0164$ and $0.0161$. Agreement across retrievers then matters more than one first-place result.
Let's consider an example.
Suppose BM25 ranks documents A, B, and C. Dense retrieval ranks B, D, and A.
Document B receives contributions from ranks two and one. Document A receives contributions from ranks one and three. Document C appears only in BM25, while D appears only in dense retrieval.
| Document | BM25 rank | Dense rank | RRF contribution with $k=60$ |
|---|---|---|---|
| A | 1 | 3 | $1/61 + 1/63$ |
| B | 2 | 1 | $1/62 + 1/61$ |
| C | 3 | Not returned | $1/63$ |
| D | Not returned | 2 | $1/62$ |
Document B wins because both retrievers place it near the top. The calculation never compares a BM25 score with cosine similarity.

RRF needs no score calibration. It can combine any retrievers that return ordered document lists.
The experiment now has three rankings to compare against under-specified probes.
Evidence that supports model-directed retrieval
The strongest case for model-directed retrieval does not come from a single retriever but rather from systems where the model can run several cheap searches and adapt after each result.
Keyword tools can recover much of a RAG baseline
An Amazon study compared conventional RAG with a tool-using agent. The agent could search documents with keyword tools but had no standing vector database.

The keyword system reached more than 90 percent of the RAG system across the reported metrics. The faithfulness result was 94.5 percent of the RAG score. Context recall reached 88.0 percent, while answer correctness reached 91.5 percent of the RAG score.
To clarify, these are relative attainments, not raw accuracy values. A 94.5 percent attainment means the keyword agent retained most of the reference RAG score on that metric. It does not mean 94.5 percent of all answers were faithful.

The mechanism is simple. A one-pass keyword query may miss. A model can read the misses, extract a better name or phrase, and search again. Several adaptive lexical searches can recover evidence that one fixed lexical query would miss.
Frequently changing corpora benefit from this design. A filesystem search sees the current text. A vector index only sees content processed by its latest ingestion job. Removing the standing index also removes chunk embedding, index storage, and synchronization work.
That saving moves the actual work to request time. Every agent search consumes latency and model tokens. A stable corpus with high query volume may still benefit from paying the indexing cost once.
Policy training improves the loop itself
The Search-R1 paper is another solution where the retriever, training data, and base model were held fixed. Reinforcement learning changed how the model used search.

The Qwen2.5-7B system improved by 24 percent over the matched RAG baselines across seven question-answering datasets. The Qwen2.5-3B system improved by 20 percent. The model learned to issue multiple queries during reasoning and use the returned evidence before answering.
Putting retrieval inside the trajectory makes those decisions trainable. The ranking function may remain unchanged while the policy improves search timing, query wording, and evidence use.
Context freshness due to tools
Anthropic's just-in-time context pattern keeps identifiers in context and loads data during the task. A path points to the current file. A stored query can read the current database state. A web link can fetch the current page.

An index can also be exposed through the same tool interface. The agent may use raw text for fresh local files and semantic search for a stable knowledge base. Model-directed retrieval does not force one storage design.
Evidence against keyword-only retrieval
The tempting conclusion is that agents made vector search obsolete. But if you holistically consider all the available evidence, there are problems with keyword-only retrieval.
Dense retrieval cannot recover details missing from the query
Semantic search is often presented as the solution to vocabulary mismatch.
A query may use billing retry while the code uses PaymentAttemptBackoff. An embedding model can place both near each other even though they share no words.

That ability has a limit. The query still needs to describe what the search should distinguish.
The CoREB benchmark shared some details on this.

CoREB evaluates code retrieval in three directions:
A text description can retrieve code, code can retrieve its description, and one code sample can retrieve related code. The result relevant here comes from its text-to-code task.
That task contains two different query formats.

- The Canonical queries are full problem descriptions.
- The Search queries are shorter, developer-style keyword descriptions averaging about 19 tokens.
Both search the same corpus of 1,744 code items, so corpus size and candidate difficulty remain fixed.
Before looking at the scores, we need to understand the metric.
nDCG@10 stands for normalized discounted cumulative gain at 10.

It evaluates the first ten retrieved items. Relevant results earn gain, but results near the top earn more than results near position ten. CoREB also uses graded relevance, so a correct solution receives more credit than a related but failed solution.
The score is normalized against the best possible ordering for that query.
- A score of 1 means the first ten results are ordered ideally.
- A lower score means useful results appear late or below weaker results.
- A score near 0 means the ranking captures almost none of the useful signal available in an ideal top ten.
Suppose the correct implementation appears at rank one. That ranking receives strong credit. If four failed solutions appear first and the correct implementation sits at rank eight, the score falls. If the correct implementation never enters the first ten, the ranking receives no gain from it.
CoREB's long Canonical queries produced nDCG@10 scores between 0.450 and 0.582 across the eleven embedding models. The shorter Search queries produced scores between 0.000 and 0.015. Even the best Search result reached only 0.015.

This is a drop of roughly two orders of magnitude. Larger embedding models did not avoid it. Code-specialized models did not avoid it either. The paper also reports that reranking did not close this gap.
The result does not show that semantic retrieval is ineffective. The same models performed far better when the query preserved the full problem description. It shows that semantic similarity cannot reconstruct requirements that never entered the query.
Consider auth flow again. An embedding model can retrieve authentication code even if the repository uses words such as identity, login, or session. But the query does not say that the destination disappears after consent.

The model therefore has no reason to prefer redirect-state code over authentication dashboards, session middleware, or login tests. All of them are semantically related to authentication.
BM25 and dense retrieval fail differently here. BM25 may miss the target because the query words never appear in it. Dense retrieval may find the correct topic but return many neighboring files because the query omits the property that separates the answer.
The engineering conclusion is specific. Do not reduce a detailed information need to a broad topic label before semantic search. Pass the full question, or build a query that preserves the component, behavior, failure, and relevant constraint. If an agent generates auth flow, the search policy has already discarded most of the information the retriever needed.
Why semantic search still belongs in the system
The CoREB result tells us not to expect an embedding model to recover constraints that the agent removed from the query. It does not remove the problem embeddings were built to solve, i.e., the words in the request and the words in the corpus often differ.

Suppose a developer asks:
Where do we retry a failed card charge?
And the relevant implementation in code might look like this:
class PaymentAttemptBackoff:
def schedule_next_attempt(self, attempt):
delay = min(2 ** attempt.count, 3600)
enqueue(attempt.invoice_id, delay=delay)The query contains retry, failed, card, and charge. The code contains none of those words.
A literal search for the full question may return documentation about payment failures, UI copy for declined cards, or nothing at all. BM25 has no direct evidence connecting retry to schedule_next_attempt or card charge to PaymentAttemptBackoff.
An agent can sometimes bridge the gap through repeated lexical searches. It might search for payment, discover PaymentAttempt, inspect nearby symbols, and then search for schedule_next_attempt.

That route works only if an early result exposes the repository's vocabulary. If the first few terms miss, the agent has nothing useful from which to reformulate the next query.
Dense retrieval offers a different first move. It embeds the natural-language request and the code or its surrounding text, then looks for items that express a similar function. Names such as backoff, next_attempt, invoice_id, and enqueue jointly describe retry behavior even though no single token matches the developer's wording.
This is the setting Cursor tested in its semantic search study. Its offline Cursor Context Benchmark compared an agent with semantic search available against an agent without it.
The agents still had traditional code-search tools, so the experiment was not semantic search versus grep in isolation. It asked whether semantic retrieval adds value when the agent can already search lexically.
Across the tested model configurations, adding semantic search increased answer accuracy by 12.5 percent on average. The improvement ranged from 6.5 to 23.5 percent depending on the model.

Cursor also ran an online A/B test because an offline retrieval task does not tell us whether developers keep the generated code. Across all requests, semantic search increased retained code by 0.3 percent. On codebases with at least 1,000 files, the increase was 2.6 percent. When semantic search was disabled, dissatisfied follow-up requests increased by 2.2 percent.
The smaller overall online gain has a straightforward explanation. Many editor requests do not need repository search. Renaming a local variable or completing the next line will not benefit from an index.
The effect becomes larger on repositories where the answer is harder to locate from the open file and its immediate neighbors.
These are Cursor's internal results, not an independent public benchmark, so they should not be treated as a universal estimate.
They still test a more useful question than "Does vector search beat grep?" The measured improvement came from giving an existing search agent another retrieval path.
Cursor's implementation also shows how semantic retrieval changes inside an agentic system. The company trains its embedding model using traces of agent sessions. An LLM reviews the files and searches that appeared later in a successful trajectory and estimates which items would have been useful earlier. Those rankings become training signal for the embedding model.
The retriever is no longer optimized only for generic text similarity. It is trained toward evidence that helps an agent complete a software task. That connects the semantic index to the control loop instead of leaving it as a frozen component in front of the model.