Continuous Profiling in Production: eBPF & OTel (2026)

Q: What is continuous profiling and how is it different from tracing?

Continuous profiling samples the call stacks of running processes at a low fixed frequency, all the time, and aggregates them so you can see which functions consume CPU or memory across a fleet. Tracing times individual requests across services. Traces tell you where latency goes; profiles tell you where compute, and therefore money, goes, including code with no span such as garbage collection, serialisation and logging.

Q: Is the OpenTelemetry Profiles signal production ready in 2026?

Not yet for critical workloads. The OpenTelemetry Profiling SIG announced that the Profiles signal entered public alpha, with Collector support from v0.148.0 and an eBPF profiling agent that runs as a Collector receiver. The announcement itself says the signal should not be used for critical production workloads yet and that production-ready backends have not emerged, so collect with open agents but keep storage reversible.

Q: How much overhead does continuous profiling add?

Sampling profilers are designed to be low overhead because their cost scales with sampling frequency and stack depth, not traffic, which is why they can stay on permanently. Exact overhead depends on the agent, runtime and frequency, and the OpenTelemetry announcement publishes no figure. Measure it on your own top services before rollout, and keep CPU profiling at a low frequency; allocation and lock profiling cost more.

Q: Should we use eBPF profiling or language SDK profilers?

Use both for different jobs. eBPF whole-system profiling sees every process on a Linux host with no code change, which makes it the right base for a fleet-wide cost ledger. Language SDK profilers give richer runtime detail and custom labels such as endpoint or tenant, so add them to the handful of services that dominate the ledger. On-demand profiling alone gives you no baseline.

Q: How do you turn profiling data into cost savings?

Run it as a ledger. Each week, aggregate CPU samples by function, convert each function's share of samples into dollars using your blended cost per core, and review the top twenty with owners. Separate the business logic you mean to pay for from pure overhead such as JSON encoding, log formatting, regex, TLS handshakes and allocation churn. File tickets with dollar estimates and verify fixes with a differential flame graph.

Q: How do profiles connect to distributed traces?

SDK profilers can label each sample with the active trace span ID, so a slow span in your tracing tool links to the stacks sampled on that thread during that span. That shows whether the span was computing, waiting on a lock or blocked on I/O. Remember that CPU profiles only show on-CPU time; for time spent waiting on downstream calls you need off-CPU or wall-clock profiles, or the trace itself.

Q: What are the security risks of continuous profiling?

The eBPF agent needs elevated kernel capabilities on every host, which is a security review in regulated or multi-tenant clusters and impossible on some managed serverless platforms. Profiles can also leak information: function names, file paths and especially custom labels may contain tenant identifiers or secrets. Apply the same redaction and access boundary you use for logs, and keep sample labels bounded and non-sensitive.

Q: When is continuous profiling not worth it?

When the fleet is small. If your total compute spend is a few thousand dollars a month, a five or ten percent saving will not cover the engineering time and backend cost of running fleet-wide profiling and a weekly ledger review. Profile on demand when there is a specific problem. It is also not worth building custom tooling on the alpha OTLP profiles wire format in 2026.

Satyam Kumar

返回博客

ai-architecture

Continuous Profiling in Production: The Profile Is the Only Telemetry Denominated in Money, So Run It as a Cost Ledger First and a Debugger Second (2026)

By Satyam KumarOctober 15, 202612 min read

continuous profiling production profiling ebpf profiling opentelemetry profiles otel profiling alpha grafana pyroscope flame graphs differential flame graph pprof cpu cost optimization cloud cost reduction observability cost span profile correlation sre performance engineering finops engineering observability 2026

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

Continuous Profiling in Production: The Profile Is the Only Telemetry Denominated in Money, So Run It as a Cost Ledger First and a Debugger Second (2026)

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment