AI Observability with OpenTelemetry: GenAI Semantic Conventions, Cost Attribution, Eval Metrics (2026)

Q: What is the AI observability pattern?

AI observability extends OpenTelemetry tracing with semantic conventions specifically for LLM workloads. Conventional observability tells you the system is up; AI observability tells you whether the system is producing useful output at acceptable cost. The pattern instruments every LLM call with the gen_ai.* attribute namespace (operation name, system, model, input/output tokens, finish reason, request parameters), adds custom attributes for cost, prompt template ID, tenant_id, feature_id, eval scores, and emits a hierarchical span structure where the agent turn is the root and every LLM call, tool call, and eval is a child. The same trace data drives operational dashboards (latency, cost), product analytics (which prompts produce best output), and offline analysis (prompt-vs-quality correlations).

Q: Why is conventional observability insufficient for AI systems?

Four reasons. Latency is necessary but insufficient — a 4.2s call producing a useful 800-token completion is fine; the same 4.2s producing 50 tokens of garbage is a disaster, and latency alone does not distinguish them. Status code is binary but LLM success is not — a 200 OK response can be a hallucination, a refusal, or a flagged-by-moderation answer; conventional observability has no schema for these. Cost attribution is invisible by default — calls range from $0.0001 to $0.10 and the LLM provider billing is divorced from trace context, making "which feature is driving our LLM bill" unanswerable. Prompt content is critical context — when an agent produces a bad output the team needs to see the exact prompt, including system message, few-shots, retrieved context, and user input; conventional observability does not capture request bodies.

Q: What are the OpenTelemetry GenAI semantic conventions?

A standardised attribute vocabulary for LLM spans that has stabilised through 2025. Core attributes: gen_ai.operation.name (chat/completion/embeddings), gen_ai.system (openai/anthropic/cohere), gen_ai.request.model (the specific variant), gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, request parameters (temperature, max_tokens, top_p), gen_ai.response.finish_reason (stop/length/tool_calls/content_filter). Custom but conventional: gen_ai.usage.cost_usd computed at emission time from token counts and a versioned cost-table; gen_ai.prompt.template_id for prompt versioning; gen_ai.eval.<name>.score for evaluation results; tenant_id, feature_id, conversation_id, turn_index for business-dimension slicing. Instrumentation libraries for major LLM SDKs now emit the standard attributes automatically.

Q: How should I structure spans for an AI agent?

The agent turn is the root span; every LLM call, tool call, retrieval call, and eval is a child. Root span (operation: agent_turn) carries user_id, conversation_id, turn_index, total cost, total latency, total tokens, final response disposition. Children carry their own LLM-specific or tool-specific attributes. The tree shape lets the team answer "what did the agent do for this user turn" by looking at one trace rather than reconstructing from scattered logs. Tool call spans carry tool-specific attributes (search query and result count, code-execution snippet and runtime, retrieval embedding ID and chunk identifiers). Span links across turns connect the current root span to the previous turn for conversation-level analysis. Flat span structure (every span a sibling) is unworkable for debugging multi-step agents — the hierarchy is the single most important architectural decision.

Q: How do I attribute LLM cost across a multi-tenant deployment?

Compute cost-per-call at span emission time using a versioned cost-table; record gen_ai.usage.cost_usd as a span attribute. Set tenant_id, feature_id, and prompt_template_id once in the agent's trace context at the start of the user turn — they propagate to every child span automatically. Aggregate cost from the trace stream in real time (or near-real-time, every 1-5 minutes) into per-tenant and per-feature buckets. Cost anomaly detection runs on the trace data: alert when cost-per-turn for a feature exceeds week-over-week 30% increase, when a tenant's cost doubles overnight, or when a prompt template suddenly produces 5× longer completions. Per-tenant cost ceilings (when contractual) are enforced by a real-time aggregator that maintains rolling costs in Redis and a request-time check that throttles or rejects when the ceiling is hit.

Q: How do I capture prompt content without exploding storage costs?

Sample full content for 1-5% of calls and always capture full content for calls that failed validation, were flagged by moderation, hit cost outliers, or exceeded latency P95. Combine sampling with redaction (NER-based PII removal plus regex for emails/phones/account numbers before storage). On every call (sampled or not), emit a prompt fingerprint — a SHA-256 hash of the normalised prompt text — so unsampled calls can still be clustered for aggregate analysis. The sampled full content enables post-hoc debugging when something goes wrong; the fingerprints enable "find all calls that used this prompt pattern" queries without storing every prompt. Storage cost stays bounded at a few % of the no-sampling cost; debugging capability is preserved for the cases that actually need it.

Q: How do eval scores fit into telemetry?

Emit eval scores as span attributes on the LLM span (sync) or as a linked span (async). When evals run synchronously after the LLM call returns and before the user response is sent, the LLM span is updated with attributes like gen_ai.eval.groundedness.score=0.87, gen_ai.eval.faithfulness.passed=true. When evals run asynchronously after the user response (so they do not block the user-facing path), emit a separate eval span that links back to the LLM span; the storage backend joins the two for queries. The result is that eval scores are queryable alongside latency and cost — "show P50 groundedness over 24 hours by prompt template" is a one-line query. Without this co-location, eval analysis requires manual joins across separate systems and eval-driven incident triage is too slow to be useful.

Q: What dashboards does every AI team need on day one?

Five. (1) Latency-and-quality: P50/P95/P99 latency by model and feature, eval score distribution alongside latency, latency drift by prompt template version. (2) Cost-per-feature: cost per turn by feature and tenant, cost-per-token efficiency, week-over-week cost trend with anomaly markers. (3) Eval score: per-eval distributions over rolling windows, per-prompt-template eval scores, user feedback correlated with eval scores, eval failure rate over time. (4) Agent-trace explorer: queryable view of full traces with filtering by user, conversation, eval failure, cost outlier, latency outlier, prompt template — the primary debugging tool. (5) Tool-call success: per-tool success rate, P99 latency, result-quality (when available), error-type breakdown — surfaces which tool to fix first when agent failures cluster.

Q: What sampling strategy works in production?

Combine head sampling (sample at trace start) and tail sampling (sample after the trace completes based on outcome). Head sampling rate of 10-100% — higher than typical HTTP services because traces are lower-volume per request and high value for debugging. Tail sampling: keep 100% of error traces, 100% of traces with eval failures, 100% of slow traces (above P95 latency), 100% of cost outliers (above 3× median cost). The remaining "normal" traces are sampled at 1-10% for storage cost control. The combination ensures every interesting trace is preserved while bulk volume is bounded. Hot retention 7-30 days for active investigation; cold retention 1 year for trend analysis and post-hoc audits.

Q: When does the AI observability pattern NOT apply?

Three scenarios. (1) Single-shot LLM features with no agent loop — a "summarise this document" feature with one LLM call needs only a single span with standard GenAI attributes, not the hierarchical structure or trace explorer. (2) Very low-traffic systems (under a few thousand calls per day) — a full OpenTelemetry pipeline (collector, storage, dashboards, alerting) is overkill; structured logs in a low-cost aggregator suffice. (3) Pre-production prototypes optimising for iteration speed — print-statement debugging plus the LLM SDK's built-in usage logging is enough until traffic and team size justify graduating; the graduation moment is usually when 2-3 engineers debug shared systems and traffic exceeds a few thousand calls per day. The pattern's overhead is justified at production scale where the trace explorer becomes essential for debugging and cost-per-feature dashboards become essential for finance.

Satyam Kumar

Back to Blog

ai-architecture

The AI Observability Pattern: OpenTelemetry Tracing for LLM Calls, Token Cost Attribution, and Eval Metrics in Production (2026)

By Satyam KumarApril 24, 202621 min read

ai observability opentelemetry genai llm tracing token cost attribution multi-tenant cost eval metrics agent trace hierarchy prompt template versioning tail sampling cost anomaly detection tool call spans production llm microservices patterns ai system design observability dashboards

The AI Observability Pattern: OpenTelemetry Tracing for LLM Calls, Token Cost Attribution, and Eval Metrics in Production (2026)

Frequently Asked Questions

Share this article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI & Cloud Architect. Helping teams build systems that scale to millions.

LinkedIn GitHub

The AI Observability Pattern: OpenTelemetry Tracing for LLM Calls, Token Cost Attribution, and Eval Metrics in Production (2026)

Frequently Asked Questions

Share this article

Comments

Leave a comment