Distributed Tracing Across Queues: Span Links Done Right

Q: Why does my distributed trace stop at Kafka or SQS?

Either the producer never injected trace context into the message headers, a relay or connector stripped the headers, or the consumer never extracted the context for each message. Fix propagation end to end first: publish one test message through every hop, including outbox relays and retry queues, and confirm the traceparent arrives intact.

Q: Should a consumer span be a child of the producer span?

Only when the work is strictly one message in, one unit of work out, processed immediately, or when the pattern is request-reply over a queue. For batches, fan-in, delayed jobs and retries, start a new trace and add a span link to the message creation context. OpenTelemetry messaging conventions use links as the default correlation mechanism.

Q: What is a span link in OpenTelemetry?

A span link records a causal relationship between a span and another span context that is not its parent, possibly in a different trace. A span has at most one parent but can carry many links, which is why links are the right tool when one consumer span processes messages produced by many different requests, as in batch consumption.

Q: Where should trace context go in a message?

In the message headers or attributes provided by the broker, not in the payload. Headers keep tracing out of your schemas and consumers, avoid changing message hashes and deduplication keys, and match what OpenTelemetry propagators and the messaging semantic conventions expect. Make sure any relay or connector copies headers forward.

Q: How do I trace a batch consumer that processes fifty messages per poll?

Extract the context from each message individually. Either create one process span per message with a link to that message creation context, or create one span for the batch with a link per message. Never extract once and attach it to the poll loop, because that makes fifty unrelated requests appear to share one trace.

Q: Why do my span links always point to missing traces?

Because each side makes an independent head-sampling decision. At 10 percent sampling on producer and consumer, both ends of a link survive roughly 1 percent of the time. Coordinate sampling: hash a business key such as an order ID into the decision on both sides, or write a sampler that honours the sampled flag of the linked context.

Q: How should retries and dead-letter replays be traced?

Treat each attempt as its own unit of work. When a retry or replay creates a new message, carry the previous attempt context forward and add it as a span link on the new process span. That keeps the causal chain from the original request through every attempt without building one enormous trace that spans hours.

Q: Are the OpenTelemetry messaging semantic conventions stable?

Not yet. In semantic conventions 1.44.0 the messaging spans page is marked Development. Instrumentations are expected to use the OTEL_SEMCONV_STABILITY_OPT_IN transition when the stable version arrives, so keep dashboards and queries that depend on messaging attribute names in one place so a rename is a single change.

Satyam Kumar

Retour au blog

architecture

Your Trace Stops at the Queue: Distributed Tracing Across Asynchronous Boundaries in 2026

By Satyam KumarNovember 2, 202613 min read

distributed tracing asynchronous tracing opentelemetry messaging span links trace context propagation traceparent header kafka tracing sqs tracing rabbitmq tracing message queue observability consumer spans batch processing tracing trace sampling w3c trace context event-driven architecture sre observability

Your Trace Stops at the Queue: Distributed Tracing Across Asynchronous Boundaries in 2026

Frequently Asked Questions

Partager l'article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

Architecte AI & Cloud. J'aide les équipes à construire des systèmes qui scalent pour des millions.

LinkedIn GitHub

Your Trace Stops at the Queue: Distributed Tracing Across Asynchronous Boundaries in 2026

Frequently Asked Questions

Partager l'article

Comments

Leave a comment