Semantic Search vs Keyword Search: Architecture and Implementation


The agent demos that survive contact with production all treat memory as an architecture, not as a context window. This is the three-tier pattern — episodic, semantic, procedural — that separates the lifecycles, read paths, write paths, and failure modes the naive single-store approach conflates. Read paths, write paths, the consolidation pipeline that earns the architecture, where each tier physically lives, eight anti-patterns to retire, the 5-stage maturity ladder, and a portable design that lands cleanly on LangGraph, AutoGen, the OpenAI Agents SDK, the Anthropic Claude Agent SDK, and the Microsoft Agent Framework.
Chunking is the highest-leverage, most-neglected decision in RAG. Fixed vs recursive vs semantic vs late vs contextual retrieval — and the pipeline that ties them together.
The 40-year-old data structure behind every search engine — and the sparse half of modern RAG. Posting lists, segments, BM25 scoring, and Lucene to Elasticsearch.
A team running a customer-support assistant on GPT-4o reported a beautiful number to their CFO: 47% cache hit-rate, $18,000 per month saved, p95 latency down from 2.4s to 380ms. Three weeks later the same team was reading a customer-trust incident report. The semantic cache had decided two refund-policy questions were "the same enough" at cosine similarity 0.93 — and 0.93 had been a reasonable threshold on the eval set, which had not contained two refund policies. The cache was not broken; the cache was working exactly as designed. The architecture was wrong. This article is the deep-architecture answer to the question the team had not asked: when does semantic cache help, and when does it lie? Four-tier hierarchy from exact-key through normalised through semantic to LLM, three-step threshold calibration using a risk-weighted decision rule, multi-tenant key design that prevents the cross-tenant-pollution incident, TTL and model-version discipline, eight failure patterns to retire, five-stage maturity ladder, and the Monday-morning checklist that ships sustainable savings without the customer-trust incidents.
Advertise and partner
Reach engineering leaders and CTOs reading about AI, mobile and cloud architecture.
Sponsored mentions and co-authored articles or technical deep-dives. Relevant products only, always disclosed.