Serverless Multi-Agent: LangGraph + AgentCore (2026)

Q: Why does LangGraph beat hand-rolled orchestration code for multi-agent production systems?

LangGraph beats hand-rolled orchestration for production multi-agent systems because of four properties that production-grade hand-rolled code eventually reinvents badly: a deterministic, reproducible execution model (every node is a pure function over state, every edge is evaluated once, traces are replayable); checkpointing and interrupt-resume as first-class operations (DynamoDB, Postgres, Redis checkpointers ship in the box, the interrupt() primitive makes human-in-the-loop tractable without you building a saga); time-travel and state introspection (rewind to any checkpoint, branch from there, replay the exact failing trace against a new prompt); and streaming-friendly state updates that compose with SSE or WebSocket transports without bespoke plumbing. The same four properties apply to LlamaIndex Workflows, AutoGen GraphFlow, CrewAI Flows, and the Microsoft Agent Framework — the architectural rule is to pick a framework that exposes a deterministic, checkpointable graph rather than to write the abstraction yourself. Hand-rolled "I will just use Python" works at the toy stage and the demo stage; it fails at the first production incident where the question is "what state was the graph in when this turn went wrong, and what did the supervisor evaluate to decide to dispatch the wrong specialist?". The cost of building this layer in-house is not the writing; it is the years of edge cases that LangGraph (and equivalents) have already absorbed.

Q: How do I choose between AWS Lambda, Bedrock AgentCore Runtime, Fargate, and Step Functions for an agent runtime?

Match the runtime to the workload shape, and accept that production estates run two or three runtimes per agent platform. AWS Lambda (or its container-image variant) is right for the orchestrator and for short specialist agents that complete in under 15 minutes — its cold-start is acceptable with provisioned concurrency, packaging is simple, REST and SSE surfaces are native, and the per-invocation cost is tiny for the typical agent payload. Bedrock AgentCore Runtime is right when you want AWS-managed integration with AgentCore Memory and Observability without doing the packaging and cold-start work yourself, and when the agent fits AgentCore’s opinionated runtime model. Fargate (or Azure Container Apps or Cloud Run with min-instances) is right for long-running agents — research crawls, multi-minute reasoning loops, agents that need persistent connections — that exceed Lambda’s 15-minute hard cap. Step Functions Standard Workflows is right when the workflow itself is the product — multi-day customer-onboarding flows, claims pipelines, document processing with human approvals — because the durability, retry semantics, and operational tooling are first-class in a way that LangGraph-in-Lambda is not. The hybrid is common and correct: Step Functions for the outer business workflow, LangGraph-in-Lambda for the agentic reasoning within each business state, Fargate for the long specialist agents the orchestrator dispatches with callbacks. Putting every workload on one runtime is the universal mistake — it produces either chronic cold-start pain (because Fargate is too slow for the orchestrator) or chronic timeout pain (because Lambda is too short for the long agent).

Q: What is the four-tier state model and why does conflating two tiers cause production incidents?

The four tiers are graph execution state (the per-run state the LangGraph runtime maintains across nodes, keyed by thread ID, lifetime = the run plus retention, access pattern = one-thread-one-writer), short-term conversational memory (within-conversation memory like "the user prefers markdown summaries", lifetime = the conversation, access pattern = read on every turn, size = a few KB to low MB), long-term episodic memory (across-conversation memory like "this user is the CTO of Acme Corp", lifetime = years, access pattern = write at conversation end, read at conversation start with vector retrieval, requires explicit forgetting and privacy controls), and long-term semantic knowledge memory (the agent’s RAG knowledge base — product catalogue, policies, regulatory rules — lifetime = years, access pattern = updated by ingestion not by conversation). Conflating graph state with episodic memory inflates the checkpoint and breaks the cost model (the checkpoint grows linearly with user history and you pay for the storage and the deserialisation on every node transition). Conflating conversational memory with episodic memory does a vector search over years of history on every turn (unbounded read amplification, latency spikes at p99). Conflating episodic memory with semantic knowledge memory creates a per-user knowledge base that pollutes other users (cross-tenant leakage is one indexing bug away). The production contract is one store per surface, with strict access boundaries: DynamoDB or Redis for graph checkpoint, AgentCore Memory or Mem0 for conversational and episodic memory (separate namespaces), a vector store for knowledge memory, no cross-writes except through explicit ingestion jobs. Every system that conflates surfaces will eventually hit one of the failure modes; the cost of separating them up front is far lower than the cost of separating them under incident pressure.

Q: When should I use a supervisor topology versus hierarchical versus swarm versus fan-out-aggregate?

Pick the topology by the shape of the work, not by what is fashionable in the framework docs. Supervisor (one orchestrator agent dispatches to specialists each turn) is right for general-purpose assistants with 4 to 8 well-bounded specialists where one model is competent as the dispatcher and the conversation has a serial shape (one specialist per turn, sometimes loop). Hierarchical (top supervisor over mid-level supervisors over specialist teams) is right when the problem decomposes naturally into bounded sub-domains and each team should have its own ownership and observability boundary — research-and-write assistants, customer-service systems with sales/support/billing teams, code-generation systems with planner/architect/coder/tester teams. The cost is depth-of-dispatch latency: three levels of supervisor calls is three model round-trips per turn. Swarm (peer-to-peer handoff with no central dispatcher) is right when the work has a pipeline shape with conditional revisits, when no agent has the global view, and when you want resilience to any single agent being unavailable — document drafting (writer hands to fact-checker hands to editor with possible revisit to writer), agentic research with multiple specialists collaborating. Fan-out-then-aggregate (orchestrator dispatches the same input to N specialists in parallel, aggregator synthesises) is right for evaluation-style workflows where independent perspectives are explicitly required for quality, for parallelism-friendly latency budgets, and when specialists do not need to converse with each other (the AWS reference architecture’s campaign review with persona reviewer + legal validator + brand checker + finaliser is the canonical example). Picking the wrong topology produces a system that is slow, opaque, or expensive regardless of how carefully you tuned the framework underneath — shape first, framework second.

Q: How does AgentCore Memory compare to LangChain memory abstractions, Mem0, Zep, and Letta?

AgentCore Memory is AWS’s managed memory service for agents, exposing both short-term conversational memory (session-scoped, low-latency K-V) and long-term episodic memory (cross-session, vector-indexed, with structured event schema) behind a single SDK. It removes the operational work of provisioning Redis for conversational memory and a vector store for episodic memory; the trade-off is AWS coupling and the opinionated event schema. LangChain’s ConversationBufferMemory, ConversationSummaryMemory, ConversationKGMemory and the newer langgraph.checkpoint integrations are framework-level abstractions that need you to provide the underlying store (Redis, Postgres, DynamoDB, in-memory) — strong for portability across clouds, weaker on the long-term episodic story because you have to assemble the vector layer yourself. Mem0 is a focused episodic-memory product (open-source core + managed cloud) with strong event-extraction-from-conversation semantics, language-agnostic SDKs, and the operational simplicity of "point your agent at it and call save/recall"; it composes well with LangGraph (you call Mem0 in a memory node before the supervisor decides). Zep is similar in shape to Mem0 with an emphasis on session-and-fact graphs and a graph database under the hood; it appeals to teams whose episodic memory has relational structure ("user X is at company Y, which uses tool Z"). Letta (formerly MemGPT) is the academic-research-derived approach to giving LLMs explicit working-memory management primitives via tool-calls; it is differentiated when the agent needs to actively manage its own memory hierarchy. The 2026 production decision is most often AgentCore Memory if you are on AWS and want the managed shape, Mem0 if you want a focused episodic-memory product across clouds, or a hand-assembled Redis + vector-store stack if you have specific requirements that the managed products do not meet.

Q: What does observability for a multi-agent system actually need to capture to be useful in an incident?

One trace per user request and one span per node execution, with the trace ID propagated from the user’s HTTP request through the orchestrator into every specialist agent and every model invocation — without this, a complaint of "the agent took 40 seconds" cannot be attributed to a slow tool, slow model, slow retrieval, or slow conditional-edge evaluation. Token usage per model call, broken down into input/output/cached, so you find the prompt that exploded to 60k tokens and is silently costing $4 per turn. Tool call inputs and outputs with content-type-aware redaction (PII redactor on the trace write path; the PII redaction pipeline pattern applies here), so incident replay is possible without storing raw user content. Conditional-edge decisions in the graph — when the supervisor decided to call the legal validator instead of the brand checker, what was the state and what value did the conditional function return? LangGraph’s LangSmith integration records this; custom observability must too. Latency decomposition per node (wall-clock baseline, then within-node retrieval/model/tool latency separately) so regressions show up before they cascade into user-visible total-latency regressions. Error rate, retry rate, and timeout rate per node — aggregated to per-node, per-hour, per-deployment, with alerts on threshold breach. Dashboards that matter from day one: p50/p99 end-to-end latency by request type, p50/p99 per-node latency, token cost per turn, cancellation rate, retry rate per node, error rate by error class, prompt-cache hit rate. AgentCore Observability + CloudWatch, LangSmith, OpenTelemetry with GenAI semantic conventions exporting to Datadog/Honeycomb/Grafana — the choice of vendor matters far less than the discipline of capturing the above on every run.

Q: How do I prevent the orchestrator from becoming a bottleneck under fan-out?

By dispatching specialists truly in parallel (not in a sequential for-loop with await), by keeping the orchestrator’s per-turn work minimal (do not preprocess in the orchestrator if you can preprocess in a node), and by accepting partial results when one specialist is slow. The LangGraph Send primitive and the Command API explicitly dispatch fan-out branches concurrently — the framework will execute them in parallel within the same process if they are CPU-cheap, or dispatch them as separate Lambda invocations if you wire the graph that way. The naive mistake is to write a for-loop that awaits each specialist sequentially, which serialises what should be parallel; the LangGraph idiom is Send("persona_reviewer", state) | Send("legal_validator", state) | Send("brand_checker", state) which produces a true parallel dispatch. For specialists that need to run on separate runtimes (because they are long-running, because they are GPU-bound, because they need different deployment shapes), invoke them as separate Lambda functions, Fargate tasks, or AgentCore Runtime instances and have the orchestrator collect their outputs via a callback URL or a shared state store; the LangGraph supervisor pattern composes with this — the supervisor node makes the dispatch and pauses, the downstream invocations write their results back, and the supervisor resumes when results are available. For latency, the aggregator should handle partial results explicitly: a 2-of-3 quorum with a clear degraded-answer return is preferable to waiting indefinitely for the slow specialist. The architectural rule is that the orchestrator coordinates and does not compute — every CPU-second in the orchestrator is paid on every turn while the per-specialist cost is paid only when that specialist runs.

Q: What is the cost model for a fan-out multi-agent system and what are the levers?

The cost model has three layers that compose multiplicatively: model token cost (input + output, multiplied by the number of model calls per turn — a fan-out with three specialists plus one aggregator is at minimum four model calls per turn before any tool-call loops); runtime cost (Lambda GB-seconds or Cloud Run vCPU-seconds during orchestration plus any Step Functions state transitions, typically 5–15% of model cost on a tuned system); and infrastructure cost (DynamoDB/Redis for state, vector store for memory, observability storage, S3 for artefacts, typically under 10% of model cost). The highest-ROI lever is prompt-cache hit rate — the same system prompt and tool definitions on every turn become free after the first call within the cache window, so structuring prompts with stable content at the front and dynamic content at the back drops cost more than any other single change. The second lever is routing cheap turns to a cheap model via the [model router pattern](/en/blog/ai-service-pattern-model-router-cost-quality-latency-aware-routing-2026) — classification turns ("is this a refund question?") do not need a frontier model. The third lever is trimming the message history before it inflates the prompt — summarisation nodes at every Nth turn keep state bounded. The fourth lever, specific to fan-out, is whether all N specialists genuinely need to run on every turn — many fan-out systems can be replaced with classify-first-then-dispatch where only the relevant 1–2 specialists run, cutting cost by 50–75%. Track cost per conversation, cost per turn, cost broken down by model and node, prompt-cache hit rate per node, and the per-tenant breakdown from day one — the architectural rule is that every dollar must be attributable to a span on the trace.

Q: How do I handle a specialist that needs to run for 40 minutes when the orchestrator is on Lambda?

Move the long specialist off Lambda and onto a runtime that supports the duration: Fargate (unlimited), Bedrock AgentCore Runtime (purpose-built for long-running agentic workloads), Step Functions Standard Workflow (one year), or Azure Container Apps / Cloud Run (60 minutes). The orchestrator stays on Lambda and uses the "dispatch with callback URL" pattern: the orchestrator invokes the long specialist asynchronously (SQS message, Step Functions task token, EventBridge event), returns immediately to the user with a job-status URL, and the long specialist posts its result back to the orchestrator (or directly to the state store) via the callback when it completes. The user-facing surface polls the job-status URL or subscribes to an SSE stream that emits status updates. The graph state is checkpointed at the dispatch boundary so the orchestrator can be replaced (Lambda concurrency cycles, runtime updates) without losing the in-flight specialist. The LangGraph interrupt() primitive is the framework-level mechanism for exactly this pattern — the graph pauses at the interrupt, the long specialist runs to completion, the graph resumes with the specialist’s result when a separate request triggers the resume. The architectural rule is that the orchestrator’s duration profile is set by the user-facing latency budget (under a few seconds for chat), and any specialist that exceeds that budget must be dispatched async with a callback. Trying to keep the orchestrator alive for the full 40 minutes by raising the Lambda timeout, switching to Fargate for the orchestrator, or holding a WebSocket open is the universal first attempt and the universal first incident.

Satyam Kumar

Back to Blog

ai-services-patterns

Serverless Multi-Agent Orchestration — LangGraph, Bedrock AgentCore, and the Architecture Pattern Behind Production AI Workflows (2026)

May 29, 202625 min read

Frequently Asked Questions

Share this article

Twitter LinkedIn WhatsApp

Satyam

AI & Cloud Architect. Helping teams build systems that scale to millions.

Serverless Multi-Agent Orchestration — LangGraph, Bedrock AgentCore, and the Architecture Pattern Behind Production AI Workflows (2026)

Frequently Asked Questions

Share this article

Comments

Leave a comment