Metric Cardinality Explosion: Cut Observability Cost

Q: What is metric cardinality and why does it matter so much?

Cardinality is the number of distinct time series your monitoring system stores, and it matters because it is the multiplicative term in your observability bill rather than an additive one. Every unique combination of metric name and label values is a separate series, each with its own storage, its own index entry, and its own memory footprint in the ingestion path. Take a single counter called http_requests_total with labels for method (7 values), status code (12), endpoint (200) and region (4). That is 7 x 12 x 200 x 4 = 67,200 series from one metric. Now add a user_id label because someone wanted to debug a specific customer, and at 100,000 users the same metric becomes 6.7 billion potential series. The code did not change, traffic did not change, the number of services did not change — one label multiplied everything by five orders of magnitude. This is why cardinality problems arrive as a step function rather than a gradual curve: the increase happens the instant a pull request merges, and the invoice arrives weeks later attached to a change nobody connects to the cost. The consequences are not only financial. High cardinality inflates index size and memory pressure, which slows queries and can lag ingestion — so dashboards become slowest during exactly the traffic spikes when you most need them, and alert evaluation can fall behind. In the worst case the monitoring system becomes an availability risk of its own. The practical rule that prevents nearly all of this: a metric label must have a bounded set of values known at design time. If the values come from user input, contain identifiers, or churn on every deploy, that dimension belongs in traces or logs, whose cost model is per-event rather than per-series-per-scrape.

Q: Which labels should never be used on a metric?

Any label whose value set is unbounded, derived from user input, or churning — because each of those produces series without limit. The clear prohibitions: user IDs, session IDs, request IDs, and correlation IDs, which are unbounded by construction and are trace data rather than metric data. Raw URL paths containing identifiers, such as /orders/8a3f-92bc, which create one permanent series per order; always template to /orders/:id at the middleware before the metric is emitted, since this usually enters through an HTTP instrumentation default rather than a deliberate choice. Free-text error messages, which vary by interpolated values and are effectively unbounded; use a bounded error_code label and put the message in the log line. Full stack traces or exception strings, for the same reason. Timestamps or any time-derived value, which guarantee a new series every interval. Email addresses, IP addresses, and device identifiers, which are both unbounded and frequently personal data you did not intend to put in a metrics store. Two categories deserve particular attention because they look safe. Pod names, container IDs and instance IDs appear bounded at any instant — there are only so many pods right now — but the storage engine retains every series it has ever observed for the full retention window, so fifty deploys a day quietly manufactures a leak that no snapshot reveals. And tenant or customer IDs are defensible only when the set is genuinely small and stable, or when you restrict the label to a bounded top-N and bucket everything else into an "other" value. The test to apply before adding any label: can you state the maximum number of distinct values it will ever have, and is that number small? If you cannot answer, it does not belong on a metric.

Q: How do I debug a specific user or request without a high-cardinality label?

Use exemplars, which is the feature that dissolves what looks like an unavoidable trade-off between aggregate metrics and individual debuggability. An exemplar attaches a sample trace ID — and optionally a few attributes — to a specific metric bucket or sample. The metric itself stays low-cardinality and fully aggregatable, but when you see a latency spike in a histogram, you can jump directly from the elevated bucket to an actual trace of a request that landed in it. You get the aggregate view and the concrete example without paying series cost for every user. This matters because the reason people add user_id to a metric is almost never that they want to graph per-user counters; it is that they want a path from "something is wrong" to "here is a request where it went wrong". Exemplars provide exactly that path at a tiny fraction of the cost, and most teams reaching for a high-cardinality label simply do not know the feature exists. Prometheus supports exemplars on counters and histograms, OpenTelemetry supports them in the metrics SDK, and Grafana renders them as clickable points on a graph. The broader principle is to let each signal do what it is good at. Metrics answer "how is the system behaving in aggregate" and must be cheap, bounded and always-on. Traces answer "what happened in this specific request" and are the correct home for high-cardinality attributes such as user, tenant, session and request ID. Logs answer "what exactly did this component report". Nearly every cardinality problem in practice is a metric being asked to do a trace’s job, and naming that explicitly tends to end the design argument quickly. If you need per-tenant aggregates for billing or SLAs, that is a legitimate case — but serve it from a small bounded top-N label, or compute it in your analytics store rather than in your real-time metrics system.

Q: How do I find out what is driving my cardinality today?

Start by measuring series counts per metric, then attribute them to owners, then find what nobody queries — in that order, because each step informs the next and the third usually pays for the whole effort. For the first step, Prometheus exposes a TSDB status endpoint and page that reports the top metrics by series count, the labels with the highest number of distinct values, and total head series; most hosted vendors expose an equivalent cardinality or usage explorer. Run it and you will typically find that a small number of metrics account for the large majority of series, which makes the problem far more tractable than it first appears. Second, attribute those series to teams or services, usually via an existing service or namespace label. Attribution is what changes behaviour: without it, cardinality is an unpriced shared resource and gets over-consumed exactly as economics predicts, and the platform team ends up arguing about an invoice with people who have no visibility into their own contribution. Third — and this is the cheapest win available — identify series that no dashboard, alert or recording rule has queried recently. In practice a large proportion of stored series are never read by anything. Deleting or dropping them at ingestion is pure saving with zero observability loss, and it typically funds the more careful work that follows. Several tools can analyse query logs and rule definitions to produce this list. Do this audit before optimising anything else. It is common to discover that the expensive metrics are ones added during an incident two years ago, kept because nobody was sure whether removing them was safe, and queried by nothing at all.

Q: Should I fix the cost by reducing retention or scrape frequency?

Those are the two most common responses and both are usually the wrong lever, because they reduce the bill by reducing the observability rather than by removing waste. Cutting retention destroys the historical comparison you depend on during an incident — the question "is this normal for a Tuesday?" needs weeks of history, and "did this regress after the release?" needs at least the interval since the release. Teams that cut retention to control cost routinely find themselves unable to answer basic questions during the next outage, which is paying nearly the same money for materially less value. Reducing scrape frequency has a subtler cost: it lowers the resolution at which you can detect short spikes, and brief saturation events simply vanish between samples. It also does far less for cost than people expect in systems where storage is dominated by the number of series rather than the number of samples per series. The correct order of operations is to remove waste first. Drop the series nobody queries, which costs you nothing. Remove or template the unbounded labels, which is where the multiplication is coming from. Move high-cardinality detail to exemplars and traces, which preserves the capability at a lower price. Only after those three should you consider tuning retention, and even then the right move is usually tiered rather than uniform: keep high-resolution data for a short window and downsampled aggregates for the long tail, so trend analysis survives while raw sample volume falls. If cost pressure is genuinely forcing a reduction in observability, that is a signal to have an explicit conversation about which questions you are choosing to stop being able to answer — a decision worth making deliberately rather than as an invisible side effect of a retention setting.

Q: How should cardinality be enforced — code review or tooling?

Tooling at the ingestion boundary, because code review structurally cannot catch this and should not be asked to. A reviewer looking at a two-word diff that adds a label has no way to compute the product of every label value set on that metric, no visibility into how many distinct values the new label will take in production, and no sense of the current series budget. Expecting them to catch it is expecting arithmetic nobody has the inputs for. Enforcement that actually holds sits in the metrics pipeline. Relabelling rules at the collector or a metrics proxy can drop known-bad label names outright, rewrite raw paths into templated routes, and cap the number of series accepted per metric so a runaway label degrades gracefully instead of taking the system down. Prometheus supports this through metric_relabel_configs, and the OpenTelemetry Collector offers equivalent processors. The key property is that these controls apply regardless of who wrote the code or whether anyone reviewed it, which is what makes them worth the setup effort. Layer three additional practices on top. Give each team a published series budget with current usage visible next to it, which turns an invisible externality into a constraint someone owns and resolves most of the problem without policy arguments. Alert on cardinality growth rate, not just absolute count, so a new unbounded label is caught within hours rather than at the next invoice. And add a lightweight design step for new metrics — an estimated series count stated the way an endpoint states a latency target — so the arithmetic happens once, at design time, by the person with the context to do it. Code review still helps as a backstop, especially with a documented list of banned label names, but it must not be the primary control.

Q: What is a reasonable cardinality budget for a team or service?

There is no universal number, and any figure quoted without context should be treated sceptically — but there is a sound method for deriving yours. Work backwards from cost and capacity rather than picking a number from a blog post. Establish what one thousand active series cost you per month in your actual setup, whether that is vendor pricing or the amortised cost of self-hosted storage, memory and query capacity. Then decide what proportion of infrastructure spend observability should represent; many organisations land somewhere in the low single-digit percentage range, but this legitimately varies with how critical availability is to the business. Divide that budget across teams by a defensible key — number of services, traffic share, or criticality — and publish both the allocation and current usage. As a sanity check on individual metrics: a well-designed metric on a typical service usually needs tens to low hundreds of series, not thousands. A single metric producing more than a few thousand series deserves a specific justification, and one producing tens of thousands is almost always carrying a label that belongs in traces. Whatever number you choose, three properties matter more than its precise value. It must be visible, so teams can see their usage against it without asking anyone. It must be attributable, so the team generating the series is the team that sees the number. And it must have consequences — even soft ones like a required review to exceed it — because a budget nobody enforces is a suggestion. The point of the budget is not to hit a particular figure; it is to make the trade-off explicit at the moment of the decision, so someone consciously chooses between the tenant label and the endpoint label instead of adding both and discovering the cost a quarter later.

Q: Does cardinality work the same way for logs and traces?

No, and understanding the difference is what makes the correct signal choice obvious. Metrics are priced per series per scrape interval: once a series exists, you pay for it continuously for the whole retention window whether or not anything ever queries it, and whether or not that label combination sees any traffic again. That standing cost is precisely why an unbounded label is so damaging — it is not one expensive event, it is a permanent tax that persists long after the request that created it. Traces and logs are priced per event. A trace carrying a user ID, a session ID, a full URL and a request ID costs the same as one carrying none of them; the high-cardinality attributes are free in the sense that they do not multiply anything. This is the entire reason the standard advice is to move high-cardinality dimensions from metrics into traces. They have their own cost dimensions — volume and retention — but volume scales linearly with traffic, which is predictable and controllable through sampling, rather than multiplicatively with the number of distinct attribute values. This asymmetry gives a clean design rule. Ask what shape the question is. "How many requests failed in the last hour, broken down by status class?" is an aggregate question with bounded dimensions, so it is a metric. "Why did this particular request fail?" is an individual question with unbounded dimensions, so it is a trace. "What exactly did this service report while handling it?" is a log. Cardinality problems are almost always the first question being asked in a way that smuggles in the second, and exemplars are the bridge that lets you move between them without paying series cost for the privilege.

Satyam Kumar

返回博客

ai-architecture

One Label Added, $40,000 a Year Gone: Cardinality Is Your Real Observability Bill

By Satyam KumarJuly 30, 202610 min read

metric cardinality observability cost prometheus opentelemetry time series monitoring sre telemetry pipeline exemplars grafana devops platform engineering finops distributed tracing system architecture 2026

One Label Added, $40,000 a Year Gone: Cardinality Is Your Real Observability Bill

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

One Label Added, $40,000 a Year Gone: Cardinality Is Your Real Observability Bill

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment