Agent Looping 2026: Prompts → Loops → Orchestrated Teams

Q: What are the three stages of agent-looping architecture in 2026 and how do they differ?

The three stages are prompt engineering (Stage 1, single prompt to single response with the human in the iteration loop, $0.001-0.05 per turn), loop engineering (Stage 2, goal to agent loop running research-draft-evaluate-improve until convergence, $0.05-2 per converged loop), and orchestrated teams (Stage 3, goal to coordinator-decomposes-and-dispatches to parallel specialists with an evaluator-agent gate, $1-50 per converged goal). They differ on five axes: unit of work (prompt vs goal-loop vs goal-team), iteration driver (human vs agent vs coordinator), failure recovery primitive (prompt versioning vs loop-breaker plus checkpoint vs per-agent breakers plus coordinator HITL), cost band, and observability requirements (per-call metrics vs trajectory trace vs multi-agent trajectory graph). The architectural significance is that each next stage answers a different question and emerges only when the previous one runs out of leverage — Stage 2 emerges when prompt-tweaking has plateaued and the workload genuinely needs multi-step retrieval or tool use, Stage 3 emerges when the goal exceeds what a single loop can hold in working context or when specialist styles must diverge. Mismatching the stage to the workload is the most common 2025-2026 failure mode — Stage 3 architectures shipped on Stage 1 problems burn 10-50x the necessary cost, and Stage 1 prompts attempting Stage 3 work produce shallow, ungrounded output that no amount of prompt-tweaking will fix.

Q: When should a workload move from Stage 1 (prompt engineering) to Stage 2 (loop engineering)?

When three signals appear together — not when one team member wants to "build agents." First, prompt iteration has plateaued: you have tried five rewrites, the eval score is stuck, and the next gain would require the model to do something the prompt cannot specify (gather missing context, evaluate its own output against criteria, retry on failure under known recovery rules). Second, the workload genuinely benefits from multi-turn reasoning: research-then-draft, plan-then-act, draft-then-critique — forcing it into a single turn either truncates the work or pushes the human into doing the multi-turn coordination by hand. Third, the output quality varies more by what the model retrieved, executed, or verified than by how the prompt was phrased — a clear signal that the architecture, not the prompt, is the bottleneck. When all three appear, advance to Stage 2 with the four production primitives that Stage 1 did not need: explicit loop budgets (step caps of 25-30 hops, token budgets, time budgets, kill switches), loop-breaker primitives (graph-depth tracker plus repeated-state-signature detector), an explicit evaluation step (separate evaluator call, never the same model self-judging), and durable execution (LangGraph PostgresSaver/RedisSaver, Temporal, AWS Step Functions, Restate, DBOS). Skipping the primitives is what produces the canonical Stage 2 incident — the loop that does not stop, runs to budget exhaustion, and ships thousands of dollars on a single failed task.

Q: When should a workload move from Stage 2 (loop engineering) to Stage 3 (orchestrated teams)?

When the goal cannot be held in a single loop’s working context, when specialist styles diverge enough that one agent persona cannot do all roles well, or when parallelism would converge faster than serial iteration. Concretely — a code-fixing loop on a 10-file repo can hold the work in one context, the same loop on a 500-file refactor cannot. A research-and-draft loop on a single topic can switch between research-mode and draft-mode prompts, a research-and-draft on three topics with strict cross-citation requirements cannot. A diagnosis loop on a single incident can hold the trajectory, a major-incident war-room with concurrent log-analysis, code-investigation, customer-impact-assessment, and remediation-drafting cannot. When those signals appear, the architecture moves to Stage 3 with the new layer of primitives: a coordinator pattern (supervisor-by-default, hierarchical for very large goals, choreographed only when emergent-discovery is the workload), an evaluator-gated merge (different model family or calibrated rubric, not the generator self-judging), durable orchestration with goal-level checkpointing, multi-agent observability with per-agent spans rolling up into a goal-level trajectory graph, and per-agent non-human identity bound to capability tokens scoping what each agent can do. The cost shape jumps from $0.05-2 per loop to $1-50 per goal — which works when the goal is genuinely worth $5-500 of human-equivalent labour and breaks when the orchestration is shipped on Stage 1 work.

Q: Why are loop-breakers the non-negotiable Stage 2 primitive, and what do they actually look like in production?

Because the default failure mode of an unbounded loop is exhaustion — the loop runs until it hits the API rate limit or the wall-clock or the budget kill-switch, and the second time it fails to converge it does the same on dozens of concurrent tasks, putting thousands of dollars on a single failed batch. A production Stage 2 loop-breaker stack has four layers. First, a step cap of 25-30 hops on the agent loop, enforced at the framework layer (LangGraph max_steps, AutoGen max_consecutive_auto_reply, custom Temporal heartbeat) — the cap is the floor, the loop should usually terminate in 5-15 steps and only hit the cap on degenerate cases. Second, a token-and-cost budget per loop, enforced by a wrapper that tracks cumulative spend across LLM calls and tool calls and short-circuits with a structured failure when the budget is breached. Third, a repeated-state-signature detector — hash the (current-tool, last-3-tool-args, current-plan-step) tuple, maintain a sliding window of the last 10, and break the loop when the same signature appears 3 times in 10 (the canonical "loop that does not stop" pattern from production swarm-AI incidents). Fourth, a heartbeat-and-watchdog at the durable-execution layer (Temporal heartbeat, Step Functions task timeout) so an agent stuck on a tool call rather than a re-planning cycle still terminates. The four layers compose — the step cap catches naive runaway, the budget catches expensive runaway, the state signature catches semantically-stuck runaway, the watchdog catches infrastructure runaway. Production agent platforms that ship without all four are the ones generating the audit-week incidents.

Q: Why must the evaluator agent be structurally separate from the generator agent in Stage 2 and Stage 3?

Because self-evaluation is blind to the model’s own failure modes. When the same model that generated the output also judges whether to terminate the loop or pass the gate, two pathologies appear consistently in production. First, self-preference bias — the model rates its own outputs higher than equivalent outputs from other sources, documented in the LLM-as-judge literature, with bias magnitudes often 5-15 percentage points on standard benchmarks. The result is loops that terminate too early because the model thinks its draft is good, and gates that pass everything because the evaluator thinks the generator did fine. Second, shared-blind-spot failures — if the generator model has a calibration error on a specific class of inputs (overconfident on adversarial prompts, hallucinating on out-of-distribution dates, miscounting in tabular reasoning), the same-model evaluator has the identical blind spot and cannot catch the failure. The production discipline is to make the evaluator structurally separate by at least one of three mechanisms: different model family (Claude generator with GPT or Gemini evaluator, or vice versa), different parameter scale or training lineage (frontier MoE generator with calibrated MLM-classifier or smaller-LLM evaluator on a narrow rubric), or fully-rule-based evaluator on critical gates (regex/schema/tool-call-success checks that are not model-judged at all). For high-stakes Stage 3 deliverables, a hybrid — model-based evaluator on quality dimensions plus rule-based gates on factual and structural dimensions — is the production-standard pattern. The same discipline applies to the eval-set itself: hold-out cases the generator was not trained or fine-tuned on, regularly refreshed, and labelled by a different process than the generator’s training data.

Q: What is durable execution and why is it required for Stage 2 loops and Stage 3 orchestrated teams?

Durable execution means the agent’s state — plan, conversation history, tool-call results, accumulated context — lives outside the worker process in a checkpointed store, so a worker crash, deployment, or scale-down does not destroy in-flight work. The four production options in 2026 are LangGraph with PostgresSaver or RedisSaver checkpointers (in-process state-machine with external persistence, ergonomic for graph-shaped agents), Temporal (workflow-as-code with deterministic replay, the dominant choice for long-running orchestrations), AWS Step Functions (managed state machine, native to the AWS stack, operationally simple), Restate or DBOS (newer entrants combining workflow durability with code-first ergonomics). The reason this is required, not optional, is that Stage 2 loops typically run for 30 seconds to 10 minutes and Stage 3 goals run for 10 minutes to several hours, and over those time spans worker crashes are not rare events — OOMs, deployments, scale-downs, network partitions, and provider rate-limit timeouts all happen at production rates that make non-durable execution a guaranteed source of partial-completion incidents. Without durable execution, a crash mid-iteration loses the entire trajectory, the user re-pays the full cost on retry, and the partial state may have left side effects (tool calls executed, payments made, emails sent) with no record. With durable execution, the state is checkpointed at every step, replay is deterministic up to the last checkpoint, side-effect tools are wrapped in idempotency-keyed calls, and resume-after-crash is the default behaviour rather than an incident response. For Stage 3 specifically, the orchestration layer must also checkpoint per-specialist results so that a coordinator restart does not re-dispatch already-completed specialist work — the cost saving on retries is dramatic on goals priced at $5-50 per completion.

Q: What is classify-first routing across the three stages and why is it the dominant cost lever?

Classify-first routing is the discipline of putting a small fast classifier (typically a 1-3B SLM at the gateway, or in the simplest cases a rule or a vector-similarity check) in front of every agent fleet, so each request is sorted by inspection into Stage 1, Stage 2, or Stage 3 before any frontier-model token is spent. The classifier produces a structured decision — stage assignment, allowed tools, allowed model class, budget cap — and the router applies it. The cost lever this gives you is the single largest in 2026 production agent economics and frequently the difference between sustainable unit economics and burn. Worked example on a 50,000-tickets-per-day support workload: running everything as Stage 2 lands at roughly $0.10-0.20 per ticket and $5,000-10,000 per day, running everything as Stage 3 lands at roughly $1-2.50 per ticket and $50,000-125,000 per day, but a classify-first fleet that puts 50% of tickets into Stage 1 ($0.005 each), 45% into Stage 2 ($0.10 each) and 5% into Stage 3 ($1.50 each) lands at roughly $0.02 average per ticket and $1,000 per day — 5-100x cheaper than blanket-stage assignment with equivalent or better quality, because the work that does not need a loop is not paying for one. The discipline extends past the initial sort: the classifier needs a feedback loop so that tickets the orchestrated team finished trivially with one specialist call demote to Stage 2 next time, and tickets where the loop failed to converge promote, with the stage-mix monitored as a primary cost SLO. Static stage assignment without the feedback loop is the slow leak that turns a $0.02 fleet into a $0.20 fleet over six months as the stage-mix drifts toward the more expensive stages.

Q: What does multi-agent observability require that single-agent observability does not?

Three additional layers on top of standard LLM observability. First, agent-identity tagging on every span — OpenTelemetry GenAI semantic conventions emit gen_ai.tool.call and gen_ai.completion spans at the LLM-call level, but in a multi-agent system you also need agent.id (the specialist), agent.role (researcher / coder / tester / evaluator), and parent.agent.id (the coordinator that dispatched it) on every span, so the trajectory graph can be reconstructed. Second, goal-level trajectory aggregation — a Stage 3 goal generates 30-200 LLM calls across multiple agents, and viewing them as a flat list is unactionable; the observability tool must aggregate spans by goal-id into a hierarchical trajectory (goal -> coordinator dispatch -> per-specialist subgraph -> evaluator gate -> result), with timing, cost, and quality metrics rolled up at each level. Third, per-agent quality and cost metrics over time — each specialist has its own performance profile (researcher accuracy on retrieval, coder pass-rate on generated tests, evaluator false-positive and false-negative rates) and these need their own dashboards and SLOs, not just aggregate goal-level metrics. The fourth implicit layer is replay tooling — because debugging a Stage 3 incident often requires replaying the trajectory to see which specialist injected the error, the traces need to be reproducible (deterministic replay against checkpointed state), which is why durable-execution is also an observability requirement, not just a reliability one. Production stacks in 2026 lean on Langfuse, LangSmith, Helicone, OpenTelemetry-native tools, and increasingly the AgentOps category of platforms that focus specifically on agent-trajectory observability rather than per-call traces.

Q: When is choreographed (decentralised) multi-agent architecture the right choice over hierarchical (supervisor) coordination?

Almost never, for the workloads most teams ship — hierarchical supervisor coordination is the production default in 2026 and choreography is an exception reserved for emergent-discovery workloads where the coordination needs are themselves unpredictable. Choreography (agents publish observations and intentions to a shared blackboard or message bus, no central orchestrator decides who runs next, the team self-organises) wins on three workload shapes: open-ended exploration where the goal is not "do X" but "discover what is here" (autonomous pentest using a stigmergic blackboard, scientific exploration agents, code-archaeology agents on unknown legacy systems), genuinely emergent-discovery research where the next investigation step depends on findings the orchestrator could not have anticipated, and adversarial workloads where the agent team must improvise around an opponent. Hierarchical supervisor coordination wins on essentially everything else — deliverable-shaped work, software engineering goals, support-ticket resolution, document drafting, customer-research, incident-response — because the supervisor pattern is dramatically easier to debug, observe, and constrain, the coordination is explicit rather than emergent, the cost is bounded by the supervisor’s plan rather than open-ended, and the failure modes (supervisor over-decomposes, specialist runs away, evaluator self-prefers) are well-understood and have known mitigations. Reaching for choreography before having tried hierarchical supervisor coordination is one of the named anti-patterns of 2025-2026 — it ships a system that is harder to debug, slower to converge, and produces lower-quality output for the same cost as a supervisor-led team. The narrow exception is when you have positive evidence that the coordination needs are emergent and supervisor-led decomposition would constrain the agents from discovering what they need to discover.