Skip to main content
67 min read

Do You Need a Vector Database?

Part 18: Whether a retrieval system needs a dedicated vector database, and the operational costs that no search benchmark shows, with implementations

👉

Recap

In Part 17, we watched a retrieval loop fill its own context window.

The loop kept every passage it read, so each round of search added to a window of fixed size.

The model reading that window used the evidence less well as it filled.

When the Context Window Fills Up
Part 17: What a retrieval loop should clear, compact and keep outside its window as its own evidence fills it, with implementations

On the loop's own passages, the 1.5B model's effective length, by NoLiMa's 85 percent rule, was 8,192 tokens, a quarter of its advertised 32,768.

Part 17 called the policy that did best under a budget recall.

It cleared old tool results from the window, kept every result in a session log outside it, and searched the log when a question arrived:

Recall clears the window, keeps the log, and searches it at answer time, from Part 17

With the 1.5B reading, that policy answered 78 percent of the planted facts, against 63 percent for the full window.

That log held 137 passages from one investigation, and we searched it in memory, so Part 17 never asked what kind of store it needs.

A production agent writes to its log at every step of every session and searches it moments later, so the log is a write-heavy store.

Part 16 named synchronisation as the main operational cost of an index.

A revised document needs reprocessing, a deleted one must disappear from the index, and an access change must reach every stored copy.

💡
This part assumes you have read Parts 16 and 17. Vector search with Qdrant comes from Parts 1 to 4, HNSW from Part 9, and BM25 and reciprocal rank fusion from Part 16. What a permission filter does to an HNSW search, and what a second copy of every document costs, are built up here.

This part asks whether those three changes are cheaper to carry with the vectors inside the database that already holds the data, or with a dedicated vector database beside it.

1

We start with what the default design leaves out and how a filter breaks a vector index, then compare four ways to store the vectors and five ways to keep them in step.

2

The hands-on then loads one million Wikipedia passages and the Halvern documents from Parts 13 to 15 and 17. Both go into Postgres with pgvector and into Qdrant, on one laptop.

It measures what search benchmarks show, the write load of an agent's memory, and how long search stays wrong after each kind of change, with seven injected faults.

With both indexes tuned, pgvector matched Qdrant's throughput at 0.95 recall against exact search, and Qdrant held its recall under narrow filters that follow content.

Under the injected faults, every wrong result came from a copy the commit's transaction did not update.


What the default design leaves out

The default design for a new retrieval system is an embedding model and a dedicated vector database.

Parts 1 to 4 followed it. Qdrant ran in Docker, and each chunk's vector went into a collection.

The chunk's text, or in Part 4 a summary of it, travelled with the vector in the point's payload. Every search result carried the passage to show the model.

For a corpus that is loaded once and then read, that design is sound.

In production, documents get revised, some have to be erased, and access to others is withdrawn.

One document, two copies

Each change lands first in the system that owns the data, usually a relational database such as Postgres. It then has to reach the copy in the vector store.

The copy holds the document's vector, its access groups, its version and, as in Parts 1 to 3, its text.

One document, two copies

Dual-write means the application writes every change to both stores.

No transaction spans the two systems, so between the commit in the database and the write to the vector store there is always a gap.

The gap between two commits

Each of these failures leaves the two copies disagreeing, and neither store reports the disagreement:

The gap between two commits
  • In a partial write, the database commits and the process dies before the vector store hears of it.
  • A delete can succeed in the database and time out in the vector store, which leaves an orphaned vector whose source row is gone.
  • A re-sync job can read rows, embed them slowly and write them back over newer versions that landed in the meantime.
  • A re-ingest without stable document identifiers adds a second copy of every document instead of replacing the first.
  • A permission change can reach the source row and never the copy, so a group that lost access can still retrieve the document.

On Monday 2 November, Halvern's Dunmoor depot opens for the season. The overflow plan for Ashcombe's volume, OVF-0421, is revised to send overflow to Dunmoor instead of Northgate.

Suppose the revision commits in Postgres and never reaches the vector store.

A search for the overflow plan still returns the old note, text included. The model reading it is told that overflow goes to Northgate.

A stale embedding is a consistency failure between two copies of one document. The copy's vector still matches the old text it was built from.

Teams that ran vector search beside Postgres have written about moving it into Postgres. Keeping two stores in sync is one of the reasons they give.

Instacart's write-up (May 2025) describes replacing a standalone FAISS service with pgvector and says "storing data in two separate systems and keeping them in-sync led to inconsistencies".

Confident AI, which replaced Pinecone with pgvector (October 2023), listed synchronisation issues beside network latency, which it called the main bottleneck.

Three properties decide the store

Whether the second copy is worth its cost depends on three properties of the workload:

  • Corpus size decides whether the vector index fits in the memory of the database the data already lives in.
  • Write rate is how often the source of truth changes, set against how soon search must reflect each change.
  • Filter selectivity is the share of the store that each query is allowed to see.
Three properties of the workload decide the store

A dedicated store can search faster at high recall and hold recall under narrow filters. It also adds a second place that every change has to reach.

In the hands-on, we measure both sides on the same data.


How a vector index works, and where filters break it

Exact search, and why an index approximates it

The simplest vector search compares the query with every stored vector and keeps the closest ones.

It is always right, and its cost grows with the corpus. On our store, that is a million dot products over 768 dimensions for every question.

An approximate nearest neighbour index narrows the search to a small part of the store. A query is compared with thousands of vectors instead of all of them.

Part 1 introduced the idea, and Part 9 walked through HNSW, the graph index that both of our stores use.

The vector database guide linked from Part 9 covers it in full:

A Beginner-friendly and Comprehensive Deep Dive on Vector Databases
Understanding every little detail on vector databases and their utility in LLMs, along with a hands-on demo.

HNSW in one figure

HNSW, from Malkov and Yashunin's paper on hierarchical navigable small world graphs, links every vector to a few nearby vectors, in layers.

The bottom layer holds every vector. Each layer above it holds a smaller random share of the vectors below, so its links cover longer distances.

A search enters at the top layer and moves greedily, to whichever linked vector is closer to the query, until none is. It then drops one layer and carries on from where it stopped.

On the bottom layer, it keeps a list of the best candidates found so far and expands them in turn. It stops when no unexplored candidate is closer than the worst one on the list.

HNSW in one figure, with layers that thin out upwards, a greedy descent, and the candidate list on the bottom layer

The walk can settle in a neighbourhood that is close to the query without holding its true nearest neighbours. Nothing then marks the results as approximate.

The settings that decide recall

Three numbers shape the graph and the search:

  • m is the most links each vector keeps per layer, and both stores allow twice as many on the bottom layer.
    • More links make the graph easier to navigate and the index larger.
  • The build list (ef_construction in pgvector, ef_construct in Qdrant) is the candidate list used while the graph is built.
    • A longer list finds better neighbours to link, and the build takes longer.
  • The search list (hnsw.ef_search in pgvector, hnsw_ef in Qdrant) is the candidate list used while searching.
    • A longer list explores more of the graph and finds the true neighbours more often, for more distance computations per query.

The two stores ship different defaults:

m build list search list
pgvector 0.8.7 16 64 40
Qdrant 1.20 16 100 the build list's value, 100, unless set

Of the three, the search list is the only one that can change from one query to the next without rebuilding the graph. It decides how far each search walks:

The search list, where a list of 40 settles after fewer steps than a list of 100 and can stop short of the nearest neighbour

Every default sets a balance between speed and recall before the store has seen any data.

ann-benchmarks, the best-known benchmark for that balance, measures recall against exact search.

For each query, recall is the share of the true top 10 that the index returns.

Experiment 1 measures it for both stores, at their defaults and across a sweep of the search list.

Published on Nov 8, 2026