CACTUS: Data Quality Gates for Production RAG

Q: What is the CACTUS framework for data quality?

CACTUS is the data-quality layer of our five-pillar production AI framework (CACTUS to SKELETS to VECTOR to SPECIALIST to CREATE), and it names six dimensions every document must satisfy before it is allowed into a retrieval index: Complete, Accurate, Consistent, Timely, Unique, and Secure. The point of the acronym is that these are engineering contracts enforced by code, monitored by automated pipelines, and measured by observable metrics — not aspirational labels in a governance deck. Complete means schema validation and mandatory field contracts, configured per source type, so headerless tables and half-parsed PDFs never reach the chunker. Accurate is the hardest to enforce programmatically because it requires ground truth, so it leans on source-authority rules, cross-system reconciliation, and statistical outlier detection. Consistent demands canonical entity resolution before indexing, because the same entity described three ways fragments a query across three regions of the vector space. Timely means every source declares a bounded staleness budget appropriate to its use case, from sub-hour for pricing to weekly for policy documents. Unique requires both exact-hash dedupe and near-duplicate detection, since boilerplate revisions pass hash checks and then crowd the top-k results. Secure treats the ingestion boundary as a security boundary: PII and secrets are classified and redacted before the embedding call, because an embedded vector is a leak you cannot easily unsee. Failing any one dimension produces symptoms that appear at the generation layer, which is why they are so expensive to debug.

Q: Why does my RAG system hallucinate even with a good model?

Because in the overwhelming majority of production cases the model is doing exactly what it was asked to do with exactly what it was given, and what it was given was wrong. A retrieval system inherits every defect in its corpus: the stale price from a document nobody re-validated, the duplicate policy revision that crowds out the current one, the table whose headers were lost at parse time, the entity that appears under three different names so the right chunk never ranks. The retriever finds the most semantically relevant chunk available and the generator faithfully renders it, complete with a citation that makes the wrong answer look sourced. The failure surfaces at the generation layer, which is precisely why teams misdiagnose it: they tune the prompt, swap the model, add instructions to be accurate, and none of it works, because none of it touches the defect. The correct debugging order is to inspect what was retrieved before inspecting what was generated. If the retrieved chunk contains the wrong fact, this is a data-quality problem and no model upgrade will fix it. In audits we consistently find that 20 to 40 percent of a corpus would fail a completeness or uniqueness check that could be written in an afternoon, in systems where the team has already spent months on retrieval tuning. Fix the corpus first: automated quality gates in front of the index turn an unfalsifiable "the AI hallucinates" complaint into a specific, measurable, fixable data defect.

Q: Should I use LLM-as-a-judge or deterministic heuristics for data validation?

Both, in a specific order, because they catch different failures at wildly different costs. Deterministic heuristics — schema validation, mandatory field checks, exact-hash and MinHash near-duplicate detection, regex and classifier-based PII scans, staleness comparisons against a declared budget — process a thousand documents for well under a dollar, at roughly 10 to 200 milliseconds per document, and they are perfectly repeatable. They cannot detect semantic wrongness, factual contradiction, or subtly poor extraction quality. An LLM-as-judge catches exactly those, and at around 2,000 tokens per document costs roughly $5 to $20 per thousand documents and takes 1 to 3 seconds each, making it 10 to 50 times more expensive and orders of magnitude slower — while introducing its own hallucination risk and non-determinism into your quality gate. The architecture that follows: heuristics gate, judges sample. Every single document passes through the deterministic checks, which run first precisely because they are nearly free and will reject the documents a judge would have been paid to evaluate. A judge then evaluates a statistical sample of what survives, five to ten percent as a defensible starting point, plus one hundred percent of anything arriving from a low-trust source. Running the judge on every document in a million-document corpus is a five-figure bill for work a hash and a schema check do better and faster, and it is a quality program that consumes the inference budget it was created to protect.

Q: How do you prevent duplicate content from degrading RAG retrieval?

With a two-stage uniqueness gate, because the naive version misses the duplicates that actually hurt. Stage one is exact-content hashing, which cheaply removes byte-identical documents ingested twice from different connectors or re-crawled without change detection. Stage two is near-duplicate detection, and it is the stage that matters: enterprise corpora are full of documents that differ by a header, a footer, a date stamp, or a single revised paragraph, and every one of them passes a hash check. Detect these with MinHash and locality-sensitive hashing over shingles for cheap lexical near-dupes, or with embedding cosine similarity above a tuned threshold for semantic near-dupes, then keep the canonical version — usually the most recent from the most authoritative source — and either drop or link the rest. The damage duplicates cause is specific and easy to observe once you look for it: they crowd the top-k. When five near-identical chunks of the same boilerplate policy occupy the retrieval slots, the one chunk containing the actual answer is pushed below the cutoff, and the model answers from boilerplate. Reranking cannot fully rescue this because the duplicates are genuinely relevant, just redundant. Deduplication belongs at ingestion, before the embedding call, both because you save the embedding cost on every discarded copy and because removing vectors from a live index later is meaningfully harder than never inserting them. Track near-dupe density in the index as a standing metric; a rising number predicts a retrieval quality decline before users report it.

Q: When should entity resolution happen in a RAG pipeline?

Before the embedding call, always, and the reason is that embeddings are where the damage becomes permanent. Consistency failures look harmless at the source: a product is called "Model X Pro" in the marketing CMS, "MXP" in the SKU database, and "Model-X-Pro" in the support ticketing system. Every one of those spellings is correct in its own system. But when all three are embedded without canonicalization, they land in three separate semantic neighborhoods of the vector space, and a user question about the product retrieves chunks from whichever neighborhood happens to be nearest — never the union of what the organization actually knows. No amount of hybrid search, reranking, or query rewriting fully reassembles what ingestion tore apart, because the fragmentation is now a property of the index geometry rather than the query. The fix is a canonical entity resolution layer that runs during ingestion: fuzzy matching, embedding-based clustering, or a dedicated master data management system maps every variant representation to a canonical form, and documents are indexed with the canonical entity attached as metadata so filtering and grouping work later. Doing this after indexing means re-embedding the affected documents, which is a backfill project rather than a configuration change. Enterprises with meaningful entity sprawl should treat the resolution layer as core infrastructure shared across the AI pipeline and the analytics stack, since both suffer identically from the same divergence.

Q: How often should data quality gates re-run on an existing index?

On a schedule matched to each source's declared staleness budget, because quality decays after ingestion and a gate that runs only once guarantees a corpus that is correct on the day it was built and progressively wrong forever afterward. The Timely dimension makes this concrete: a document that satisfied a sub-hour freshness budget in January violates it by July, and nothing about the document changed — the world did. Practically, every source declares two things at onboarding: a staleness budget appropriate to its use case (sub-hour for pricing data, daily for inventory, weekly or monthly for policy documents), and an owner. A scheduled re-validation job then re-runs the CACTUS gates against indexed content on a cadence tighter than the budget, and a document failing re-validation is evicted from the index rather than left to answer questions with obsolete facts. Eviction requires the same machinery as embedding-lifecycle management and re-indexing, which is why the two systems should be designed together rather than bolted on separately. Accuracy re-validation deserves its own cadence, driven by upstream change signals: when the authoritative system for a data type publishes a change, dependent indexed documents are flagged for re-check rather than waiting for the calendar. Track p50 and p99 age of indexed content per source against its declared budget as a standing dashboard metric; a source whose p99 age is drifting upward is telling you a connector has quietly broken before any user notices a wrong answer.

Q: What metrics prove that RAG data quality gates are working?

Three, tracked per source, and a fourth that ties the whole thing to user-visible outcomes. First, quarantine rate broken down by dimension: what fraction of incoming documents fail each of the six CACTUS gates. This number is diagnostic rather than something to minimize — a source that suddenly quarantines thirty percent when it historically quarantined two has changed upstream and nobody told you, and catching that within a day is the entire value of the metric. Second, staleness distribution: the p50 and p99 age of indexed content per source, measured against that source's declared budget, which catches broken connectors and abandoned pipelines long before a user reports an outdated answer. Third, near-duplicate density in the index, which predicts retrieval quality decline because duplicates crowd the top-k and push real answers below the cutoff. Fourth and most important, groundedness on a fixed evaluation set: the fraction of generated answers whose claims trace back to retrieved chunks, alongside retrieval metrics like recall@k and precision. Frameworks such as RAGAS package the standard retrieval and generation metrics so you are not inventing them. The discipline that makes these numbers useful is the diagnostic habit they enable: when groundedness falls, the first question is not "which model" but "which gate," and the quarantine and staleness dashboards usually answer it within minutes. A quality gate you cannot measure is a quality opinion, and opinions do not survive contact with a production incident.

Q: Is data quality more important than model choice for enterprise AI?

For retrieval-based systems, yes, and the asymmetry is larger than most teams expect. Model capability improved dramatically through 2026, and for the overwhelming majority of enterprise RAG applications it stopped being the binding constraint some time ago; what limits answer quality now is the context the model receives, which is a direct function of what is in the index and how it got there. The counterintuitive consequence is that a more capable model can make data quality problems worse in perception, because it renders wrong source material more fluently and more persuasively, complete with a citation. There is also an economic argument. Swapping models is a configuration change measured in hours; improving data quality requires organizational process change — declaring source owners, agreeing staleness budgets, building entity resolution, funding a quarantine review queue — and is therefore slow, unglamorous, and permanently underfunded relative to its impact. That gap is exactly why data quality is the highest-leverage investment available to most enterprise AI programs, and why we place it as the foundational layer of the five-pillar framework rather than a preprocessing footnote. The practical test: before the next model evaluation, measure what fraction of your corpus would fail a completeness or uniqueness check. If that number is above ten percent, the model comparison is measuring noise, and the engineering hours would return more if spent on the gates in front of the index.

Satyam Kumar

Назад к блогу

ai-architecture

The CACTUS Framework: Automated Data Quality Gates for RAG

By Satyam KumarJuly 10, 202610 min read

cactus framework rag data quality data validation pipeline rag hallucination llm as a judge entity resolution deduplication pii redaction ingestion pipeline data quality gates groundedness ragas great expectations production rag ai architecture patterns context validation 2026

The CACTUS Framework: Automated Data Quality Gates for RAG

Frequently Asked Questions

Поделиться статьёй

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI & Cloud архитектор. Помогаю командам строить системы, масштабируемые до миллионов.

LinkedIn GitHub

The CACTUS Framework: Automated Data Quality Gates for RAG

Frequently Asked Questions

Поделиться статьёй

Comments

Leave a comment