Agent Memory Architecture (2026): Episodic, Semantic, Procedural Three-Tier Pattern

Q: Why is "context window plus vector store" not a memory architecture?

Five compounding reasons. The context window is a working buffer, not a store — anything truncated to fit the token budget is silently gone for the next turn, with no error and no observability. A vector store over conversation chunks retrieves at the wrong granularity (raw turns out of context), with the wrong salience (semantic similarity beats temporal recency for most agent tasks the wrong way around), and with no notion of "this fact was true on Tuesday but the user updated it on Thursday." There is no procedural memory at all — the agent learns nothing across sessions, every customer who asks the same question gets the same first-attempt answer, every workflow runs from scratch. The write path is implicit — what gets remembered is whatever happened to flow through the conversation history; what gets forgotten is whatever happened to fall out of the window; there is no design intent so there is no debuggability. The privacy and tenancy boundary is undefined — a vector store keyed by conversation ID composes badly with multi-tenant SaaS, with right-to-be-forgotten under DPDP / GDPR / Privacy Act, and with team-versus-user memory scopes. The cumulative effect is an agent that is unpredictably reliable, expensive to operate, and impossible to evolve because there is no separation of concerns to evolve along.

Q: What are the three tiers and what does each one store?

Episodic — time-ordered records of what happened in a session: user messages, agent responses, tool calls and results, decisions made, errors encountered. Lifecycle: session, with configurable retention (days to months). Storage: a transactional store (Postgres, DynamoDB) keyed by (tenant, user, session, turn) with a structured schema, plus an embedding index for similarity. The episodic store is also the agent's audit log. Semantic — facts, entities, relationships, preferences, profile data; the abstracted knowledge that survives across sessions. Lifecycle: months to permanent with explicit invalidation. Storage: a property graph for relationships, a document store for entity profiles, a vector index for unstructured fact passages, all keyed by (tenant, user-or-team, fact-id) with provenance pointing back to episodic events. Procedural — workflows that worked, tool-argument shapes that succeeded, decision policies refined from experience. Lifecycle: long-lived, version-controlled, sometimes promoted to deterministic code. Storage: a workflow registry, a tool-call playbook, a policy registry. The three tiers are physically separate, schematically distinct, and individually addressable; the retriever is the only component that touches all three; the consolidation pipeline is the only component that writes from episodic to semantic and procedural.

Q: What is the consolidation pipeline and why does it matter?

Consolidation is the scheduled (or threshold-triggered) job that reads recent episodic events and produces three outputs: extracted facts (entities mentioned, candidate facts about them, with insert / update / retire operations on the semantic store, all with provenance), detected workflows (sequences of tool calls and decisions that achieved a goal, accumulated as candidates for procedural promotion), and summarisation (older episodic events not promoted get summarised into a session digest that replaces raw turns in retrieval to keep the token budget bounded). The consolidation runs out-of-band of the agent turn so it adds zero latency to the hot path. It is operationally a small service with a scheduled trigger plus an event-driven trigger (on N new episodic events for a tenant, or on a tenant's quiet period beginning). Its eval suite is the central control on memory quality — when consolidation degrades, the agent's perceived intelligence degrades within a few sessions. The consolidation pipeline is the second-most-important component in the system after the retriever and gets the same operational rigour: SLOs, evals, observability, version control, ability to backfill on improvement.

Q: How does the retriever rank and compose memories from the three tiers?

At the start of each turn, the retriever runs a ranked composition: episodic recency (last N turns of this session, deterministic), semantic relevance (top-K facts about the entities mentioned in the current turn, scored by embedding similarity plus freshness), procedural match (top-M workflows whose preconditions match the current goal, scored by similarity to the current task). The composition is shallow on purpose — assemble the working buffer in 50-200 milliseconds, not run a research project. Per-tier scoping is mandatory — every retrieval call carries explicit tenant ID and privacy scope; storage refuses queries without a tenant filter; this becomes a build failure rather than a runtime leak. Freshness over similarity for facts that change — semantic facts have an as_of timestamp; mutable fact types apply a freshness filter before similarity ranking; a two-month-old preference loses to a one-week-old one. Provenance on every retrieved memory — every fact carries the chain back to the episodic event that produced it (visible to the model so it can self-correct, visible in the audit log so an analyst can trace belief). Result envelope is hard-capped — when the cap is hit, lowest-ranked items are dropped explicitly and logged; the agent sees a known-quality assembled context rather than one that grew until something timed out.

Q: Why must the write path go through consolidation rather than direct semantic writes from the turn loop?

Because direct writes from the turn loop conflate three different kinds of write with three different lifecycles, three different evidence requirements, and three different failure modes — and once they are conflated, the architecture loses the properties that make it valuable. Episodic writes are per-turn, structured, synchronous, with the agent considering the turn incomplete until the write succeeds; they are the source of truth and need transactional guarantees. Semantic writes are derived from accumulated episodic evidence, need provenance back to the episodes that produced them, need confidence and sample size, need invalidation rules; doing this from the turn loop produces facts of unknown quality with no audit trail and no ability to roll back when the extraction logic improves. Procedural writes need evidence of repeated success across diverse contexts before promotion; promoting from a single episodic instance produces brittle workflows that fail on the second invocation. The consolidation pipeline is the chokepoint that enforces the right cadence and the right evidence requirement for each tier; bypassing it is the most common architectural mistake and the most expensive one to undo, because the resulting semantic store has unknown provenance and cannot be safely re-derived.

Q: How is right-to-be-forgotten handled across the three tiers under DPDP / GDPR / Privacy Act?

Deleting a user's memory is a single API call that fans out across all three tiers within a few hundred milliseconds. Episodic — transactional delete keyed by (tenant, user) over the structured store and the embedding index; the audit log retains a redacted shell (timestamp, action, no content) where the regulatory regime requires retention of the deletion event itself. Semantic — delete facts whose only provenance was the deleted user's episodic events; for facts that have provenance from multiple users (team-shared facts derived in part from this user), reduce confidence by the lost-evidence weight rather than delete (the fact still exists; the deleted user's contribution is gone). Procedural — workflows that included the deleted user's decisions but whose success criteria are met by other users' evidence remain; workflows whose only evidence was this user are demoted to candidates and pruned. The architecture is designed so this operation is fast, complete, and auditable rather than a multi-day batch job that misses some embeddings — the per-user index in each tier is a first-class design constraint, not an afterthought. The deletion API emits an audit event with the count of records affected per tier, the regulator-facing evidence is in the same audit log, and the tenant-facing confirmation surfaces in the product as a regulatory artefact.

Q: What are the eight anti-patterns to retire when adopting this architecture?

The single-vector-store memory — all memory in one vector store keyed by chunk ID; no tier, no provenance, no scoping, no freshness. The conversation-history-as-memory pattern — "memory" is the message list passed to every model call; no long-term store, no consolidation, no cross-session learning. The implicit write path — whatever happens to land in the message list becomes "memory"; no design intent, no debuggability. The session-only memory — the agent forgets everything when the session ends; no promotion to long-term tiers. The cross-tenant vector store — no tenancy in the query path, relying on filter-after-retrieve; the first leak is permanent. The freshness-blind retriever — ranks by similarity only, ignores as_of; stale facts beat fresh ones deterministically. The unobservable consolidation — a summarisation job that runs in the background with no metrics, no eval, no audit trail; memory quality drifts and nobody notices for weeks. The procedural-memory-as-prompt — "skills" are encoded by stuffing more instructions into the system prompt; the prompt grows monotonically, the eval suite cannot catch regressions across skills, cost-per-turn grows with the skill library. Retire each and replace with the corresponding tier and discipline.

Q: What is the maturity ladder and where do most production agents sit in 2026?

Stage 0 — Conversation history only: no memory beyond the message list; demo-quality. Stage 1 — Conversation history plus a vector store: the "RAG over chat" pattern; single tier, no consolidation, no scoping; most early production agents in early 2026 sit here. Stage 2 — Episodic store with structured schema: per-turn structured events written to a transactional store with tenancy; audit log is solved; long-term memory is not; some teams have shipped this. Stage 3 — Episodic plus semantic with consolidation: facts about entities live in a structured store; consolidation pipeline runs scheduled; freshness and confidence are first-class; the agent stops forgetting things across sessions; this is the threshold at which agent reliability stops being a recurring complaint. Stage 4 — Three-tier with procedural promotion and demotion: procedural memory is live; workflows promote and demote based on observed success; the agent's skill library grows and prunes itself; the consolidation pipeline has its own evals and SLOs; this is the threshold at which the agent gets demonstrably better month over month rather than holding steady. Most production agents in early 2026 are at Stage 1 with some teams at Stage 2; the investment to move to Stage 3 is the highest-payback architectural decision a team building agents will make this year.

Q: How does the three-tier pattern compose with frameworks like LangGraph, AutoGen, the OpenAI Agents SDK, and the Microsoft Agent Framework?

The three-tier pattern is provider-agnostic and runtime-agnostic; the agent runtime is the consumer of the retriever and the producer of episodic events, but the three stores and the consolidation pipeline live outside the runtime. In LangGraph, the retriever is a node that runs at the start of each step and assembles state; the episodic store is written by the state-checkpointing layer (or a thin wrapper around it); the consolidation pipeline is an out-of-band job, not a graph node. In the OpenAI Agents SDK or the Anthropic Claude Agent SDK, the retriever wraps the agent invocation and injects retrieved memories into the system prompt; the episodic store is written from the SDK's event hooks. In AutoGen or CrewAI, the retriever is a tool the agent invokes (or, more reliably, a prompt-time injection that runs before agent execution); the episodic store is written from the orchestrator. In the Microsoft Agent Framework, the retriever is a middleware that runs on every agent step. The pattern survives migration between frameworks because the tiers are not coupled to the runtime — moving from one framework to another is a change in the retriever's callsite, not a redesign of the memory architecture. This portability is the main reason to invest in the architecture as a separable concern rather than coupling it to whatever framework is in fashion this quarter.

Q: How do I build this incrementally without a six-month rewrite?

Build the episodic store first — two engineering weeks. Define the structured schema, write the per-turn event from the existing agent code path (a thin wrapper around the LLM call is enough), stream to a transactional store with tenancy, retain for an initial 30-90 day window. Now you have the audit log; you can already debug agent behaviour better than 90% of teams. Add the consolidation pipeline second — two to four weeks. Start with summarisation only (older turns get summarised into a session digest, the digest replaces raw turns in retrieval); this is the cheapest and highest-leverage consolidation step and removes the unbounded-context-cost failure mode immediately. Promote facts to the semantic store third — four to six weeks. Add a property graph or document store for entity facts, write the fact-extraction job, populate from a backfill of existing episodic events, wire the retriever to compose semantic facts with episodic recency. The agent stops forgetting things across sessions. Promote workflows to the procedural store fourth — ongoing. Identify the top 10-20 workflows your agent runs, instrument success criteria, accumulate evidence, promote when thresholds are met. The skill library grows from production traffic rather than from a separate authoring effort. Each step ships value independently; no step requires the next step to be useful; the architecture is a destination but the sequence is a path.

Satyam Kumar

返回博客

ai-architecture

Agent Memory Architecture: Episodic, Semantic, Procedural — the Three-Tier Pattern (2026)

By Satyam KumarMay 11, 202620 min read

agent memory ai agent architecture episodic memory semantic memory procedural memory three tier memory memory consolidation agent state langgraph memory agent retriever long term memory agent reliability multi tenant agent right to be forgotten ai architecture ai agents 2026

Agent Memory Architecture: Episodic, Semantic, Procedural — the Three-Tier Pattern (2026)

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

Agent Memory Architecture: Episodic, Semantic, Procedural — the Three-Tier Pattern (2026)

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment