Tool-Calling Schema Design for LLM Agents (2026 Production Pattern)

Q: What is the single highest-ROI change I can make to my agent’s tool-calling layer?

Replace every exception-throwing tool runtime with a typed error envelope of the shape `{ok: false, code, retryable, hint, evidence}`, with the eight canonical codes (auth_failed, quota_exceeded, validation_failed, not_found, conflict, timeout, upstream_failed, internal). In a recent production audit of a B2B SaaS agent at ~3M turns/month, this single change dropped average turns-per-resolved-task from 4.7 to 2.9 (39% blended cost reduction) with no model change, no prompt change, and one engineer-week of work. The reason it works is that exceptions serialise stack traces and provider-specific terminology into the model’s context, triggering guess-loops where the model invents tool calls that don’t exist; typed envelopes give the model a structured signal it was trained to repair from, with a hint field that names the corrective action. The runtime catches every exception at the tool boundary, maps it to one of the eight codes, fills in a human-readable hint, and serialises only the envelope — the model never sees a Python traceback. Pair this with a hard repair-loop cap (one retry per tool call, then user-visible error) to prevent runaway retry loops, and the cost ceiling is bounded by design.

Q: How do JSON Schema, Pydantic, and zod differ in practice for tool definitions, and which should I pick?

Pydantic v2 is the dominant Python choice because the pipeline `Model.model_json_schema()` → per-provider transformer → tool registry is mature, and the validation code on the response path is the same code as the schema export. Zod 3.23+ with zod-to-json-schema is the dominant TypeScript choice; the major footgun is preferring `z.discriminatedUnion` over `z.union` because the former emits clean `oneOf` with a discriminator that OpenAI strict-mode accepts, the latter emits `anyOf` that the model picks among randomly. Hand-written JSON Schema 2020-12 is the path of last resort, necessary only when the tool surface crosses language boundaries and the lingua franca is the schema itself. The mistake to avoid in all three is leaving `additionalProperties` unspecified — under OpenAI strict mode that’s a 400-error at registration time, but under Anthropic and Gemini it silently allows the model to invent fields the tool will reject at runtime. Always set `additionalProperties: false` and let the canonical schema (Pydantic or zod) be the source of truth, with a per-provider transformer that emits the dialect each provider actually parses.

Q: How does parallel tool calling work across providers, and what safety contract do I need?

OpenAI defaults `parallel_tool_calls: true`, Anthropic defaults `disable_parallel_tool_use: false`, and both will happily emit four parallel tool calls when the user asks "check weather in London, Paris, Berlin, and Tokyo". The latency win is real (four parallel 200 ms calls finish in 200 ms wall-clock instead of 800 ms) and so are the failure modes. You need three guarantees. First, race-safe side-effects: every mutating tool takes an `idempotency_key: string` parameter the model is required to populate, server-side de-duplicated with a 24-hour key window. Second, partial-failure semantics: when 3 of 4 calls succeed and 1 fails, the runtime returns a mixed array of payloads and error envelopes, and the system prompt instructs the model to act on the successes and surface the failure to the user without invalidating the successes. Third, fan-out caps: hard per-turn limits (typical: 8 for read-only, 3 for mutating, 1 for destructive), enforced in the runtime before invocation, with typed envelopes returned in place of over-cap calls so the model can re-plan. The decision rule is: enable parallel on read-only, disable on mutating unless idempotent, never enable on cross-tool sequences where later arguments depend on earlier results.

Q: What does "description engineering" mean for tool schemas, and how much accuracy does it actually buy?

Description engineering is the discipline of treating every `description` field in the schema as a few-shot instruction to the model, because that’s exactly how the model reads it. Four rules pay back in measured accuracy. First, name units in fields: `amount` invites the model to pass a string or float, `amount_usd_minor` (integer USD cents) gives unit + type in one signal and cross-currency confusion drops to zero. Second, enum-gate everything with a fixed domain: a status field with `enum: ['pending','confirmed','cancelled']` is parsed correctly across providers, a `status: string` field is filled with whatever the model thinks reads well (including `"confirmed."` with trailing period or `"Pending"` capitalised). Third, examples per field not per tool: Anthropic and Gemini both reward parameter-level `examples: [...]` arrays; OpenAI strips the keyword but the model still reads it because of training-data cross-pollination. Fourth, tool names are verbs in imperative: `search_customer_by_email` outperforms `customer_search` by roughly 15 percentage points on tool-pick accuracy in measured eval sets because the imperative verb cues the model that this is an action.

Q: Why is the per-provider transformer non-negotiable, even if I only ship to one provider today?

Because the providers normalise lossy-differently, and the moment your roadmap calls for a second model (typically: a cheaper fallback, a faster regional model, or a frontier model for the hard 5% of queries) the schema that worked on the first provider does not work on the second. OpenAI strips `format` keywords and collapses `oneOf` to `anyOf`. Anthropic folds nested `$ref` inline and ignores `description` on enum members. Bedrock Converse drops `default` values silently. Gemini rejects `additionalProperties:false` outright and uses an OpenAPI 3.0 closed-schema flag instead. A Pydantic model exported via `.model_json_schema()` carries keywords several providers silently drop. The architectural answer is one canonical schema (Pydantic v2 or zod), a per-provider transformer that emits the dialect each provider actually parses, and a contract test that round-trips a known-good payload through each provider’s tool-call path on every merge. Build the transformer when you build the second provider integration, not after the first production incident where the model never picks a tool on Gemini that it picks reliably on Claude.

Q: How should I version tool schemas in production, and what counts as a breaking change?

Tools are APIs and they need semver. Breaking changes are: parameter rename, parameter removal, enum-value removal, type narrowing (string → enum, number → integer, optional → required), or adding a `required` field. Non-breaking changes are: parameter addition with `required: false`, enum-value addition, description edit, hint refinement, and example addition. The architectural rule is that breaking changes ship under a new tool name (`search_customer_v2` alongside `search_customer`) with a deprecation window of 30–90 days during which both names resolve and both are observed in the registry; this gives in-flight conversations a safe path and gives cached prompts time to refresh. Non-breaking changes ship under the same name with a patch-version bump. The enforcement layer is a JSON-Schema diff CI gate (use `json-schema-diff` or a hand-written walker against the canonical Pydantic models) that runs on every PR touching the tool layer, classifies the change, and fails the build if a breaking change is shipped without a name change. Skip this gate and the first time you find out a deploy broke production is when the model starts returning the `validation_failed` envelope on every turn.

Q: What is "lazy tool loading" and why does it matter for cost?

Lazy tool loading is the pattern of putting only the candidate tools for a turn into the prompt, not the full catalog. A 50-tool catalog with 4 KB of description per tool is roughly 50,000 schema tokens added to every prompt; at OpenAI `gpt-4o` input rates ($5/M) that is $0.25 per turn of pure schema tax before the model does any work, and on a 10M-turn/month workload that is $2.5M per year. The mitigation is a small classifier (regex, embedding similarity, or a 1.5B-parameter intent model) that runs first, picks 5–10 candidate tools for the turn, and emits only those into the prompt. The cost arithmetic flips: 10 tools × 4 KB = 40,000 tokens saved per turn, $0.20 saved per turn, $2M saved per year against a typical $2k/month classifier infrastructure cost. The latency cost is 10–40 ms for the classifier pass, which is invisible against the 1–2 second LLM round-trip. Combined with prompt caching applied to the stable classifier-emitted tool definitions (input billed at $1.25/M cached vs $5/M uncached), the compounded saving on a 10M-turn workload reaches 95% from the naive baseline with no model change and no quality regression.

Q: What OpenTelemetry attributes should I emit on every tool-call span?

The OpenTelemetry GenAI semantic conventions (draft 2024, stable 2026) define the canonical shape. Span name: `gen_ai.tool.call`. Required attributes: `gen_ai.tool.name` (the tool that was invoked), `gen_ai.tool.call_id` (the model-emitted ID, used to correlate the call with its response), `gen_ai.tool.args_hash` (sha256 of the JSON-stringified arguments — recording verbatim leaks PII; the hash gives correlation without exposure), `gen_ai.tool.result_code` (one of the eight envelope codes plus `ok`), `gen_ai.tool.retry_count` (number of repair attempts before success or final failure), `gen_ai.tool.duration_ms` (wall-clock execution time), and `gen_ai.tool.parallel_batch_size` (1 for serial, N for parallel batches). The parent span is the model invocation (`gen_ai.model.invoke`); the grandparent is the user turn (`gen_ai.turn`). The dashboards that come out of this shape are tool-pick accuracy per tool name, repair-rate per tool, error-envelope distribution per tool, and parallel-batch latency speedup — all the metrics you need to operate the agent.

Q: How do capability tokens work, and when do I need them versus a simple per-tenant allow-list?

Capability tokens are the runtime authorisation layer that sits between "this tenant is entitled to call this tool" (the allow-list) and "this user granted consent for this specific invocation pattern" (the token). For read-only tools (search, fetch, lookup) the allow-list is usually sufficient — if the tenant’s plan includes the customer-search feature, the model can call it freely within rate limits. For destructive tools (`delete_customer`, `cancel_subscription`, `transfer_funds`) the allow-list is necessary but not sufficient because tenant-level entitlement does not equal user-level consent. The capability token pattern is short-lived (typical: 5 minutes), single-use, and minted only after a human-in-the-loop confirmation step the agent surfaces in chat ("I’m about to cancel subscription #12345; reply CONFIRM to proceed"). The token rides in the tool call as an additional required parameter, the runtime validates it against a Redis-backed mint-and-burn registry, and the tool refuses to execute if the token is missing, expired, or already redeemed. The pairing with the OWASP LLM Top 10 is direct: capability tokens are the architectural mitigation for LLM06 (excessive agency), and the broader pattern is documented in the human-in-the-loop escalation pattern.

Q: What’s the five-stage maturity ladder for tool-calling, and where do most teams stall?

Stage 0 is free-form text — no tools, the agent answers from training data, production failures look like hallucinated facts. Stage 1 is ad-hoc JSON — each tool has a hand-written schema, no canonical source, no CI gate, no error envelope; tools throw exceptions; the "model keeps calling it wrong" ticket recurs. Stage 2 is typed canonical schema — Pydantic v2 or zod as source of truth, per-provider transformer, `additionalProperties: false` everywhere, error envelopes on the top-5 tools. Stage 3 is the versioned tool registry — semver enforced via JSON-Schema diff CI gate, per-tenant allow-lists in the resolver, error envelope on every tool with the eight-code taxonomy, idempotency keys on every mutating tool, per-turn fan-out caps, OTel `gen_ai.tool.call` spans with the full attribute set. Stage 4 is the lazy-loaded, prompt-cached, eval-gated registry — classifier-first routing emits 5–10 candidate tools per turn from a 50+ catalog, prompt caching on stable tool definitions, golden eval set in CI on every schema change with tool-pick accuracy and repair rate gated at named thresholds, capability tokens via HITL on destructive tools. Most teams stall at Stage 1 or Stage 2 because the cost of not progressing is invisible — the schema tax doesn’t show up as a line item on the invoice, the runaway retry loops look like model failures, and the on-call hours look like normal SRE work. The progression takes 8–16 weeks of focused engineering for a team of two to three.

Satyam Kumar

Back to Blog

ai-services-patterns

Tool-Calling Schema Design for LLM Agents — The 2026 Production Pattern

May 28, 202623 min read

Frequently Asked Questions

Share this article

Twitter LinkedIn WhatsApp

Satyam

AI & Cloud Architect. Helping teams build systems that scale to millions.

Tool-Calling Schema Design for LLM Agents — The 2026 Production Pattern

Frequently Asked Questions

Share this article

Comments

Leave a comment