Vector Index Tuning: HNSW, IVF & Product Quantization

Q: What is the difference between HNSW and IVF vector indexes?

HNSW (Hierarchical Navigable Small World) and IVF (Inverted File) are both approximate nearest-neighbour indexes but they organise vectors differently. HNSW builds a multi-layer proximity graph where each vector is a node connected to its nearest neighbours, and a query searches by greedily hopping through the graph from an entry point toward the closest vectors. It delivers the best recall at a given latency and is the default for low-latency, high-recall serving, but it stores a navigable graph on top of the full-precision vectors, so it is memory-hungry and slow to build. IVF instead partitions the vector space into clusters using k-means at build time, and at query time it only searches the few clusters nearest the query, skipping the rest. IVF builds faster and uses less memory, and its speed is directly tunable through how many clusters it probes, but at equivalent recall it is usually a little slower than HNSW. The practical rule: choose HNSW when latency and recall are paramount and memory is available, and IVF when build speed, memory footprint, or very large corpora dominate.

Q: What is efSearch and how do I tune it?

efSearch is HNSW's query-time parameter that sets the size of the dynamic candidate list the search keeps as it traverses the graph. A larger efSearch explores more of the graph before returning, which raises recall (you are less likely to miss a true nearest neighbour) but increases latency because more nodes are visited and more distances computed. A smaller efSearch is faster but can drop relevant results. Crucially, efSearch is a query-time dial, so you can change it per request or per workload without rebuilding the index, unlike M and efConstruction which are fixed at build time. The way to tune it is to assemble a labelled query set with known correct neighbours, then sweep efSearch across a range such as 40, 80, 160, 320, and 500, measuring recall@10 and p95 latency at each point. Plot the resulting curve and choose the smallest efSearch that meets your recall target — often 95 percent — within your latency budget. If no value satisfies both, you must raise M and rebuild the graph to shift the whole curve.

Q: What is Product Quantization and when should I use it?

Product Quantization (PQ) is a compression technique for vectors that dramatically reduces memory at the cost of some accuracy. It splits each vector into several equal-length sub-vectors, runs k-means on each sub-vector position to learn a small codebook of representative centroids, and then stores each sub-vector as just the one-byte id of its nearest centroid. A 768-dimension float32 vector that occupies about 3,072 bytes can compress to roughly 96 bytes, a 32x reduction, which is what lets a billion-vector index fit in RAM instead of requiring far more expensive memory or spilling to disk. The downside is that distances computed on the compressed codes are approximate, so raw PQ recall is lower than full-precision search. You should use PQ when your corpus is large enough that storing full vectors in memory is too costly, typically tens of millions to billions of vectors, and you should almost always pair it with a re-ranking stage: over-fetch a larger candidate shortlist using the fast approximate PQ distances, then re-score that shortlist with the full-precision vectors to recover most of the lost recall. For small corpora PQ is unnecessary and just adds error.

Q: How do I measure recall for a vector index?

Recall measures how many of the true nearest neighbours your approximate index actually returns, and you cannot tune an index responsibly without it. The method is to build a ground-truth reference once: take a representative sample of real queries, and for each compute its exact top-k nearest neighbours using a flat (brute-force, exact) index over the same vectors. That exact result is your truth. Then for any approximate configuration you want to evaluate, run the same queries and compute recall@k as the fraction of the true top-k that the approximate index returned — for example, if the exact top-10 and the approximate top-10 share 9 vectors, recall@10 is 0.9. Measure recall together with p95 and p99 latency, not just average latency, because a configuration that looks fast on average can have a long tail that breaks SLAs under load. Re-run this measurement whenever the corpus changes substantially or you switch embedding models, because the vector distribution shifts and a previously well-tuned index can quietly degrade. Treat the recall-versus-latency curve, not a single number, as the object you are tuning.

Q: Do I even need an ANN index for my corpus?

Not always. Approximate nearest-neighbour indexes exist to make search sub-linear on large corpora, but they add memory overhead, build time, configuration complexity, and an accuracy trade-off, so they are not free. If your corpus is small — roughly under a million vectors, though the exact threshold depends on dimensionality, hardware, and latency budget — a flat (exact, brute-force) index is often fast enough, and it has the advantage of returning perfect results with no recall to tune. Many production systems with tens or hundreds of thousands of vectors run perfectly well on exact search and would gain nothing but complexity from HNSW or IVF. As the corpus grows into the millions and brute-force latency exceeds your budget, that is the signal to adopt an ANN index, starting with HNSW or IVF, and only reaching for Product Quantization when full vectors no longer fit affordably in memory. The right default is to start exact, measure when it stops meeting your latency SLA, and add approximation only when the data forces it.

Q: Which index parameters require a rebuild and which are live?

This distinction matters because rebuilds are expensive and live parameters let you adapt instantly. For HNSW, M (the number of edges per node) and efConstruction (the build-time candidate list size) are set when the graph is constructed and changing either requires rebuilding the entire index; efSearch, by contrast, is a query-time parameter you can change on every request without any rebuild, making it the dial you use to trade recall against latency in production. For IVF, nlist (the number of clusters) is fixed at build time because it is determined by the k-means partitioning, and changing it means re-clustering and rebuilding; nprobe (how many clusters to search per query) is a live, query-time dial. Product Quantization codebooks are learned at build time, so changing the compression scheme requires a rebuild. The practical implication is to set the structural parameters — M, efConstruction, nlist, PQ configuration — deliberately at build time based on your corpus size and memory budget, and then do your day-to-day recall-versus-latency tuning purely through the live dials, efSearch and nprobe, which need no rebuild.

Q: How does index choice affect RAG answer quality?

Index choice affects answer quality because the retriever can only pass to the model the chunks it actually finds, and an under-tuned index silently drops relevant results before the reranker or language model ever sees them. If your index runs at 80 percent recall, then on roughly one in five queries a genuinely relevant passage is missing from the candidate set, and no amount of prompt engineering, reranking, or model upgrading can recover information that was never retrieved. This is why index tuning is one of the cheapest and highest-leverage quality levers in a RAG system: raising recall from 80 to 95 percent often improves answers more than a far more expensive model change. It also explains a common failure pattern where teams spend months iterating on prompts and swapping models to fix wrong or incomplete answers, when the real cause is a default-configured index quietly operating at low recall. The correct sequence is to measure retrieval recall first, tune the index to a target like 95 percent within the latency budget, and only then attribute remaining quality problems to chunking, reranking, or the model.

Q: How do HNSW and PQ work together for very large corpora?

For very large corpora you often want both the low query latency of a graph index and the small memory footprint of compression, and combining HNSW with Product Quantization gives you that. In this configuration the navigable graph structure of HNSW is built over PQ-compressed vectors rather than full-precision ones, so the index keeps HNSW's fast graph traversal while storing each vector in a few bytes instead of a few kilobytes, which lets a large corpus fit in memory. The trade-off is the same approximation that plain PQ introduces: distances computed on compressed codes during traversal are coarse, so recall is lower than full-precision HNSW. As with IVF plus PQ, the remedy is a two-stage search — use the compressed index to retrieve an over-sized candidate shortlist quickly, then re-rank that shortlist using full-precision vectors held in a smaller store to restore most of the lost recall. This pattern is how engines serve hundreds of millions to billions of vectors at interactive latency without the cost of keeping every full-precision vector in RAM, and it should be evaluated against a ground-truth set just like any other configuration because the right shortlist size depends on your recall target.