Telemetry Pipeline Architecture: OTel Collector Design

Q: What is a telemetry pipeline?

A telemetry pipeline is the layer between the applications that emit traces, metrics and logs and the backends that store them. In OpenTelemetry it is usually built from Collector instances that receive data, process it (sample, redact, enrich, filter) and export it to one or more destinations. Its purpose is to make decisions about telemetry once, centrally, before you pay to store it.

Q: What is the difference between the OpenTelemetry Collector agent and gateway patterns?

In the agent pattern a collector runs next to the application, as a sidecar or one per node, and receives telemetry locally; it is the right place to add host and Kubernetes metadata and buffer briefly. In the gateway pattern a central, horizontally scaled pool of collectors receives telemetry from many agents or services. Most production fleets use both: agents for context, gateways for sampling, redaction and routing.

Q: Why does tail sampling require all spans of a trace on the same collector?

Tail sampling decides whether to keep a trace after it completes, based on outcomes such as errors or latency, so the deciding collector must see every span of that trace. The tail sampling processor documentation states all spans for a given trace must be received by the same collector instance. With round-robin load balancing each replica sees fragments and makes wrong decisions, so a trace-ID-aware routing layer is required.

Q: How does the load-balancing exporter help with tail sampling?

The load-balancing exporter in OpenTelemetry Collector Contrib routes telemetry consistently by a routing key. If no key is configured it routes traces by trace ID, so every span of a trace goes to the same downstream collector. Teams place it in a first, stateless layer that only routes, in front of a second, stateful layer that runs the tail sampling processor, and scale the stateful layer carefully.

Q: In what order should OpenTelemetry Collector processors run?

Start with the memory_limiter processor, which its documentation recommends as the first processor so backpressure reaches receivers before data is dropped. Then apply redaction and attribute hygiene, enrichment from trusted sources such as a service catalogue, tail sampling for traces, and finally batching with a persistent sending queue so a backend outage causes delay rather than data loss.

Q: Does a collector gateway remove observability vendor lock-in?

It removes the transport lock-in, which is the expensive part: applications export OTLP to a local agent, so changing or adding a backend is a gateway configuration change rather than a redeploy of every service. It does not move dashboards, alert rules, saved queries or runbooks, which still live in the vendor product. Budget for rebuilding those when you plan a migration or negotiate renewal.

Q: Should PII redaction happen in the collector or in the application?

Both, with different roles. Redaction at emit time, where the log line or span attribute is created, is the primary control because it stops sensitive data from leaving the process at all. The collector gateway is the enforced backstop: one versioned policy that catches what emitters miss and lets you prove an attribute is never stored. Relying only on the gateway turns compliance into one regex list.

Q: When should I not build an OpenTelemetry Collector gateway?

Skip it when you run a few services with one backend, when nobody can own it on call, when you hope it will fix poor instrumentation, or when your work is mostly long-running batch workflows that tail sampling handles badly. Start with node-local agents, which give most of the decoupling benefit cheaply, and add a gateway when cost, PII control or a second destination becomes a real problem.

Satyam Kumar

返回博客

architecture

Telemetry Pipeline Architecture: The Collector Tier Is Where Observability Decisions Get Made

By Satyam KumarOctober 26, 202613 min read

telemetry pipeline opentelemetry collector otel collector gateway otel collector agent tail sampling load balancing exporter observability cost reduction vendor lock-in observability trace sampling strategy pii redaction telemetry log routing otlp kubernetes daemonset collector memory limiter processor observability architecture sre platform engineering

Telemetry Pipeline Architecture: The Collector Tier Is Where Observability Decisions Get Made

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

Telemetry Pipeline Architecture: The Collector Tier Is Where Observability Decisions Get Made

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment