Agent-Level Circuit Breakers: Per-Tool, Per-Provider, Per-Capability (2026)

Q: What is the agent-level circuit breaker pattern?

Agent-level circuit breakers replace the conventional single circuit breaker with a three-layer hierarchy — per-tool, per-provider, and per-capability — each tuned to the failure mode it is meant to protect against. AI agents fan out to a heterogeneous set of tools (search, retrieval, code execution, function calls), to multiple LLM providers (one for reasoning, another for embeddings, a third for cheap classification), and to internal capabilities (planning, validation, response generation). Each dimension has its own failure profile, latency distribution, cost per call, and consequence on the agent's task. A single breaker treats them all the same and is wrong almost everywhere — it either trips so coarsely that one slow tool kills the whole agent, or it trips so loosely that one runaway dimension exhausts the budget. The three-layer hierarchy treats each dimension on its own terms.

Q: Why are conventional circuit breakers wrong for AI agents?

Conventional breakers were designed for a world where one service calls one downstream and failure modes are mostly binary (available or not). AI agents inhabit a different world — many heterogeneous failure axes within a single user turn. When wrapped around the whole agent, the breaker tells you something failed but not which subsystem; tool failures and LLM provider failures need different fallback responses (alternative tool vs equivalent provider vs skip-and-mark); per-tool failure rates have different acceptable baselines (web fetch normally fails 6%, vector DB normally fails 0.1%); and budgets are spent unevenly across tools so a uniform breaker misses cost or latency budget exhaustion. A single breaker cannot encode three different fallback strategies; per-dimension breakers can.

Q: How do I calibrate breaker thresholds per dimension?

Failure-rate thresholds depend on the baseline. A useful starting heuristic is "trip at 3-5× the normal baseline failure rate over a rolling window of 50-200 calls". Web fetch with a 6% baseline trips at 18-30%; vector DB with a 0.1% baseline trips at 0.3-0.5%; LLM provider with a 0.5% baseline trips at 1.5-2.5%. Window length matters — 10 calls is too noisy, 1000 is too slow to react; 50-200 is the sweet spot. Add latency thresholds (trip when P99 exceeds 3× normal for 30+ seconds) to catch soft slowdowns where calls succeed but consume the budget. Add cost thresholds for LLM calls (trip when cost-per-call exceeds 5× normal sustained over 5 calls) to catch runaway prompts. Review baselines quarterly as tool behaviour evolves.

Q: What fallback strategies belong at each layer?

Tool-layer fallbacks are alternative-source-or-graceful-absence: search → cached search; vector DB → keyword search; primary code sandbox → backup sandbox; or proceed without the tool and mark the response ("could not retrieve current data"). Provider-layer fallbacks are equivalent-provider-with-same-prompt: send the same prompt to the secondary provider, transparent to the rest of the agent. Capability-layer fallbacks are skip-the-capability-mark-the-response: when validation trips, the response goes out with "validation skipped"; when planning trips, the agent falls back to a simpler answer-then-tool loop. Every tool, provider, and capability must register its fallback handler when added to the agent; code review enforces "no breaker without a fallback".

Q: Why do I need latency-based and cost-based tripping, not just failure-based?

A tool that has gone from 200 ms median to 8 second median is in active degradation even though success rate is still 100%; the breaker should trip on the latency change. Watch P99 latency on a rolling window and trip at "P99 exceeds 3× the normal P99 for 30+ seconds". Cost-based tripping is unique to agents — an LLM call with a runaway prompt (context grew to 50K tokens) costs ten times normal; the breaker counts it as a successful call and the agent burns through its budget on a broken conversation. Trip when cost-per-call exceeds 5× normal sustained over 5 calls. Conventional services do not have variable per-call cost so cost tripping does not apply; for agents it is one of the most useful additions over conventional breakers.

Q: How should retries compose with agent-level breakers?

Retries inside a closed breaker, never inside an open breaker. When the breaker is closed (normal operation), the agent may retry transient failures with backoff. When the breaker is open, retries are suppressed entirely — the call short-circuits to the fallback. The mistake to avoid is retrying above the breaker and inflating the failure count: 3 retries on one underlying problem are counted as 3 failures, the breaker trips spuriously. The right composition is: the breaker counts unique user-facing call attempts (not retries within an attempt); retries happen below the breaker; the breaker trips on the user-facing error rate, not the raw downstream error rate. Per-call timeouts (5-30s for LLMs, 200ms-2s for tools) are mandatory — without them the breaker has no signal because hung calls are counted as in-flight rather than failed.

Q: How does the half-open state work for LLM providers?

Conventional breakers use a single probe call during half-open to decide whether to close. For LLM providers and expensive tools, a single probe is not enough — the failure rate during outage is rarely 100% so a single successful probe can prematurely close the breaker, causing it to immediately re-trip. Allow 5-10 probes through during the half-open window and require > 80% success rate before closing. Add jitter to probe timing (random 10-30% of the open window) to avoid thundering-herd re-tripping when many agent instances share the same provider breaker. The trade-off is slower recovery; the benefit is far fewer false-recovery flips. For high-traffic multi-instance deployments, share breaker state via Redis so one instance tripping informs the others.

Q: When does an agent need shared breaker state across instances?

When traffic is high enough that per-instance breakers waste many calls before tripping. With ten agent instances each independently observing a provider degradation and each independently taking 50 calls to trip, the provider sees 500 wasted calls before any breaker opens. Shared state via Redis or a similar fast store lets one instance's observation inform the others, so the breaker trips after roughly 50 calls in aggregate rather than 500. The trade-off is the operational complexity of distributed state (consistency, latency, failure modes of the state store itself). For low-traffic deployments (under ~10 calls per minute per dimension) per-instance breakers with appropriate jitter are sufficient; for high-traffic agent platforms, distributed state is worth the complexity.

Q: What does the breaker dashboard look like in production?

A real-time view showing every breaker in the three-layer hierarchy with its current state (closed / half-open / open), recent trip history, current rolling-window failure rate, P99 latency, and cost-per-call. Per-tool breakers show baseline vs current; per-provider breakers show primary-vs-fallback routing percentage; per-capability breakers show which capabilities have been degraded recently. Alerts fire on (1) any breaker open for > 60 seconds, (2) sustained primary-to-fallback routing > 20%, (3) capability-level breaker trips because they directly affect user experience. The dashboard surfaces degradations in the LLM ecosystem before customers notice them — typically the first signal of a provider-side issue is a per-provider latency P99 climb, often minutes before the provider's status page acknowledges it.

Q: When does the agent-level circuit breaker pattern NOT apply?

Three scenarios. (1) Single-tool, single-provider agents — an agent with one LLM provider and no tools has no fan-out to protect; a per-call timeout plus retry-with-backoff is sufficient. Use the conventional circuit breaker pattern when traffic warrants. (2) Very low-traffic agents (below ~10 calls per minute per dimension) — rolling-window statistics are too noisy; a single failure represents 10% of the window. Use longer wall-clock windows (10-30 minutes) and conservative thresholds, or skip breakers entirely and rely on per-call timeouts. (3) Strictly-deterministic batch pipelines — failed inputs can be quarantined and reprocessed manually rather than via real-time fallback; per-call timeouts and retry-with-backoff are usually sufficient because there is no user waiting. Add breakers when the batch starts running near a deadline budget where degraded throughput would cause missed SLOs.

Satyam Kumar

返回博客

ai-architecture

The Agent-Level Circuit Breakers Pattern: Per-Tool, Per-Provider, Per-Capability Isolation in Production AI Agents (2026)

By Satyam KumarApril 24, 202622 min read

circuit breaker ai agent resilience per-tool breakers per-provider breakers multi-provider fallback llm provider routing agent observability graceful degradation cost-based tripping latency-based tripping half-open probing distributed breaker state microservices patterns ai system design production llm

The Agent-Level Circuit Breakers Pattern: Per-Tool, Per-Provider, Per-Capability Isolation in Production AI Agents (2026)

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

The Agent-Level Circuit Breakers Pattern: Per-Tool, Per-Provider, Per-Capability Isolation in Production AI Agents (2026)

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment