Vector Database Architecture: Choosing & Scaling (2026)

Q: Should I use pgvector or a dedicated vector database?

Start with pgvector if you already run Postgres and hold under a few million vectors. It keeps your embeddings transactionally consistent with the relational data they describe, makes metadata filtering ordinary SQL, and adds zero new systems to operate — a real advantage because a second datastore is permanent operational debt. With an HNSW index, pgvector comfortably serves millions of vectors at low latency for most applications. Move to a dedicated store — Qdrant, Weaviate, or Pinecone — when a named constraint forces it: you need billions of vectors, you need metadata-filtered search at low latency that Postgres cannot sustain, you need horizontal sharding and replication you do not want to operate by hand, or you want a fully managed serverless option. The wrong reason is "we might need billions someday"; design for the scale you have plus a foreseeable horizon, not a hypothetical one, and graduate when filtered-search latency, vector count, or ops scale actually demands it.

Q: What actually breaks first as a vector database scales?

Almost never raw similarity search — it is filtered search, index memory, and recall degradation. Filtered search is the classic trap: queries that must find the nearest neighbours that also match a tenant, date, or permission. Naive systems retrieve the global top-k by similarity and then drop rows that fail the filter, so a tenant with little data can get zero results because the global top-k belonged to other tenants, and in multi-tenant systems this also leaks one tenant's vectors into another's results. Memory breaks next: HNSW indexes are RAM-resident and large, so cost and instance sizing are driven by vector count times dimensions times index overhead, independent of query volume. Recall breaks most insidiously: pushing ef_search down for lower latency silently reduces recall, which surfaces as "the RAG got worse" with no error and no alert. Correct systems filter during search, capacity-plan memory up front, and track recall as an explicit SLO so degradation is visible rather than silent.

Q: What is HNSW and how do its parameters affect performance?

HNSW (hierarchical navigable small worlds) is the dominant approximate-nearest-neighbour index: it builds a layered graph that gives excellent recall and low latency, at the cost of being memory-resident and memory-hungry. Three parameters govern its behaviour. M is the graph connectivity (neighbours per node): higher M improves recall but increases memory and build time. ef_construction is the candidate-list size during index build: higher values build a better-quality graph but take longer to construct. ef_search is the candidate-list size at query time: higher values raise recall but increase latency. A common production starting point is M around 16 and ef_construction around 128, with ef_search tuned to hit roughly 95–98% recall at a p95 latency under about 50 milliseconds. The dangerous knob is ef_search: lowering it makes queries faster while silently dropping recall, so you must measure recall against a labelled set rather than tuning latency blind. Where lower memory matters, IVF and disk-based indexes trade some recall and latency for far smaller footprints.

Q: Why is "search-then-filter" a problem, and what is the fix?

Search-then-filter retrieves the top-k most similar vectors globally and only afterward removes those that fail your metadata filter (tenant, date, access control). It produces two serious failures. First, correctness: a tenant or category with relatively little data can receive empty or sparse results, because the global top-k was dominated by other tenants' vectors and the post-filter discarded everything. Second, security and isolation: it means the search engine is computing similarity across all tenants and only the application layer is supposed to filter, so any bug — or a similarity attack crafted to match another tenant's content — surfaces cross-tenant data. The fix is to filter during search: the database restricts the candidate set by the filter before or while computing similarity, so results are both correct for sparse tenants and structurally isolated. Qdrant payload filters, Weaviate filtered vector search, Pinecone namespaces, and pgvector composite indexes all support this. In multi-tenant systems treat filter-during-search as a security boundary, not just a relevance optimisation.

Q: What drives vector database cost, and how do I reduce it?

Cost is dominated by memory, not compute, because HNSW indexes live in RAM. A rough model: a 1536-dimension float32 vector is about 6 KB raw, so 10 million vectors is roughly 60 GB before index overhead, and HNSW adds about 1.5 to 2 times that, landing near 100 to 120 GB resident — which sets your instance size and bill regardless of how many queries you run. Three levers cut this hard. Quantization (scalar, product, or binary) shrinks vectors by 4 to 32 times for a modest, measurable recall hit. Dimensionality reduction, smaller embedding models, or Matryoshka-style truncatable embeddings cut the base vector size. Disk-based indexes (DiskANN-style) trade higher tail latency for much cheaper storage when you hold billions of vectors and can tolerate it. Managed serverless options price on storage plus reads and remove the memory-sizing exercise at a per-unit premium, while self-hosting puts the memory math, and the savings, back in your hands. Whichever you pick, model cost per million vectors explicitly and tier cold data rather than keeping everything in expensive RAM.

Q: How do I handle multi-tenancy in a vector database?

Multi-tenancy in vector search is both a correctness and a security problem, and the right approach depends on tenant count. The non-negotiable is that tenant scoping happens during search, not after, so one tenant's vectors can never appear in another's results even if an application call site has a bug. For a moderate number of tenants you can use payload or metadata partitioning with database-enforced filters; for very high tenant counts, namespaces or partition keys scale far better than a dedicated index per tenant, which only scales to hundreds, not hundreds of thousands. Beyond the search boundary, remember that embeddings of personal data inherit the residency and privacy rules of that data, and that embeddings can be partially inverted to recover sensitive content, so isolation, encryption, and access control matter at the vector layer just as they do at the database layer. Treat per-tenant filtering as a hard boundary with synthetic cross-tenant probes in monitoring that expect empty results and alert on any leak, exactly as you would test any tenancy isolation control.

Q: Do I still need reranking if my vector search is good?

Yes. Approximate-nearest-neighbour search is, by definition, approximate: to hit production latency it gives up some precision relative to an exact search, and the top-k it returns is a fast estimate rather than the true best matches. A reranking stage — typically a cross-encoder that scores each candidate against the query more expensively but more accurately — recovers the precision the ANN approximation traded away, reordering the top-k so the genuinely most relevant results rise to the top before they reach the model or the user. This matters most when recall has been tuned aggressively for latency, because reranking compensates for a lower ef_search. The standard production shape is therefore a two-stage retrieve-then-rerank pipeline, often combined with keyword (BM25) search for hybrid retrieval, sitting behind a cache hierarchy so the hot path stays fast. Trusting raw ANN top-k as final results throws away precision you can cheaply recover, so reranking is part of a well-built vector retrieval system rather than an optional extra.

Q: What is a sensible maturity path for vector search infrastructure?

Stage 0 is embeddings in a flat file or array scanned linearly, which is fine for a prototype but falls over past tens of thousands of vectors. Stage 1 is pgvector alongside the application database with an HNSW index and SQL filters — the right home for most teams under a few million vectors, with one fewer system to run and transactional consistency with real data. Stage 2 moves to a dedicated store (Qdrant, Weaviate, or Pinecone) for filtered search at scale, with recall tracked as an SLO and a reranking stage on the hot path. Stage 3 adds sharding and replication, quantization to control memory, filter-during-search enforced as a tenancy boundary, and embedding-version discipline so you never blend incomparable vectors. Stage 4 models cost per million vectors, tiers cold data to disk or quantized indexes, places data with residency awareness, and gates change on recall, latency, and memory dashboards. The common mistake is reaching for a dedicated database a stage early on the strength of benchmarks; teams that scale smoothly stay on pgvector until a named constraint forces the move, then choose on filtering and operations rather than raw throughput numbers.