Exact over approximate

There’s a reflex among people building retrieval systems to reach for an approximate nearest-neighbour index from day one. FAISS makes IndexIVFFlat look like the grown-up choice and IndexFlatL2 like the toy. For most of the RAG systems I’ve actually shipped, that instinct is backwards. I default to exact search and treat the move to approximate as a decision I have to justify with numbers, not assume.

What the two actually do

IndexFlatL2 is brute force. For each query it computes the true distance to every vector and returns the real nearest neighbours. No training, no parameters, no surprises:

index = faiss.IndexFlatL2(dimension)
index.add(embeddings)
distances, ids = index.search(query, k)   # exact top-k, always

IndexIVFFlat trades exactness for speed. It runs k-means over your vectors to carve the space into nlist cells, and at query time it only looks inside the nprobe cells nearest the query. That skips most of the distance computations, and can skip the cell your true nearest neighbour happens to live in. It also needs a training step on a representative sample before you can add anything, and two hyperparameters you now own forever.

The cost nobody prices in

The reason I’m cautious isn’t speed. It’s that approximate search fails silently. When nprobe < nlist, the index can quietly miss the most relevant passage, and nothing tells you. Your pipeline still returns five chunks, the model still writes a confident answer, and the answer is just a little worse than it should have been. In an auditable system, where the whole point is that an answer is grounded in the best available evidence, a retrieval layer that occasionally drops the best evidence is the exact failure mode you can’t afford. (I have more to say about bugs that return a confident, wrong result.)

Exact search has no such mode. The top-k it returns is the top-k. When something is wrong, it’s wrong in your chunking or your embeddings, where you can see it.

Where the crossover actually is

So when is approximate the right call? When the numbers force it. Two things drive that:

• Latency. Brute force is O(n·d) per query. With 384-dimensional MiniLM vectors, a modern CPU chews through tens of thousands of vectors in well under a millisecond and low millions in single-digit milliseconds. Document-scale RAG (one contract, one manual, even a whole knowledge base of a few thousand chunks) never gets near the latency wall.
• Memory. A flat index is n × d × 4 bytes. At 384 dims that’s ~1.5 KB per vector, so a million vectors is ~1.5 GB sitting in RAM. This is usually what actually pushes you off flat: the index stops fitting comfortably before queries get slow.

Put together, the honest crossover for typical embedding sizes sits in the low millions of vectors. Below that, exact search is faster to reason about, returns ground truth, and deletes a category of bugs. Above it, you reach for IVF (or HNSW), you measure recall against a flat baseline, and you tune nprobe until the recall loss is one you can defend.

The rule

Start exact. Keep a flat index as your ground-truth oracle even after you scale, so you can measure what an approximate index is costing you in recall. Only go approximate when a profiler, not a reflex, tells you to. Most systems never get there, and that’s fine. Correctness you can see beats performance you can’t measure.