AI Broke Your DORA Dashboard: What to Measure Now

Q: Did AI-assisted development actually make DORA metrics worse?

The delivery metrics got worse in the data even as developers felt more productive, and both facts are true at once. Google 2024 DORA report found that a 25% increase in AI adoption was associated with an estimated 1.5% decrease in software delivery throughput and a 7.2% decrease in delivery stability, despite 75.9% of respondents using AI for part of their work and roughly 75% reporting productivity gains. The key is that these measure different things. Individual productivity — how much code a developer produces and how fast they feel — went up. Team delivery performance, which is what the four DORA keys measure, flattened or declined. They diverged because AI made the writing of code cheap without making the review, testing, and integration of that code any cheaper, so the extra output piled up at the parts of the pipeline that did not speed up. The 2025 DORA report reframes this as AI being an amplifier: it magnifies the strengths and weaknesses of the delivery system it lands in. A team with strong review and small batches gets faster; a team without them ships more code through the same weak gate and gets less stable. So AI did not break the four keys — the four keys correctly reported a regression that individual-productivity dashboards were hiding.

Q: Why does AI-assisted coding increase change failure rate?

The dominant mechanism is batch size. DORA research has consistently shown that larger changesets carry more risk, because a bigger diff is harder to review thoroughly, more likely to contain an unnoticed defect, and harder to reason about when it fails. AI makes it trivially easy to produce more code in a single change — you can generate a large feature, a broad refactor, or many files at once in the time it used to take to write a fraction of that. So the natural consequence of AI adoption, unless you actively counter it, is that changeset sizes grow. Bigger changesets are riskier deploys, and riskier deploys mean a higher change failure rate. This is why the regression shows up in stability metrics specifically rather than throughput: the code gets written and merged quickly, but a larger share of those changes break something. There is a secondary effect too. About 39% of developers in the 2024 report expressed some distrust in AI-generated code, yet the code still ships, and if it is reviewed less carefully because reviewers are overwhelmed by volume, or tested with AI-generated tests that assert what the code does rather than what it should, the failures pass through the gate. The countermeasure is not to reject AI but to keep batch sizes small, review at depth, and make change failure rate a headline metric so the regression is visible before production.

Q: Which metrics should I stop reporting now that we use AI coding tools?

Stop reporting the creation-side metrics as measures of progress: lines of code, commit counts, pull requests merged, and AI acceptance rate. Every one of these measures the part of the software pipeline that AI made cheap, which means they went up for a reason that has nothing to do with whether more value was delivered safely. Before AI, these were rough proxies — writing code was the constraint, so more code loosely tracked more delivery. AI severed that link. Now generation is nearly free, so lines and commits measure activity, not outcomes, and reporting them up-and-to-the-right actively misleads leadership into believing output equals delivery. AI acceptance rate is the most seductive trap because it looks AI-specific and sophisticated, but it only tells you how often developers accept suggestions, not whether the resulting software is better, faster to ship, or more stable. Replace them with metrics that measure delivery and risk: the four DORA keys (deployment frequency, change lead time, change failure rate, failed-deployment recovery time), plus batch size as a leading indicator and review latency split out of lead time. The test for any metric in the AI era is simple — if it can go up purely because code generation got faster, it is no longer a performance signal and should not be on a performance dashboard.

Q: What are the four DORA metrics and are they still valid in 2026?

The four key DORA metrics are deployment frequency (how often you deploy to production), change lead time (how long from code committed to running in production), change failure rate (what proportion of deployments cause a failure requiring remediation), and failed-deployment recovery time (how quickly you recover when a deployment fails). They remain valid and useful in 2026 — the argument that AI broke DORA is really about which metrics people watch, not about the four keys being wrong. In fact the four keys are more valuable now, because they are exactly the metrics that caught the delivery regression AI introduced while individual-productivity dashboards missed it. What changes with AI is emphasis and decomposition. Change failure rate deserves promotion to a headline metric because it is the signal AI pressures hardest through inflated batch sizes. Change lead time should be split into time-writing versus time-in-and-waiting-for-review, because when writing collapses toward zero, review latency becomes most of the lead time and is invisible in the aggregate. And you should add a metric the classic four do not include — batch or changeset size — as a leading indicator, since it predicts change failure before the deploy happens. So the honest 2026 position is: keep the four keys, read them as delivery signals rather than individual output, elevate change failure rate, decompose lead time, and add batch size.

Q: How do I measure the review bottleneck that AI creates?

Instrument review as its own throughput, which starts with splitting change lead time into two components: the time spent writing and committing code, and the time spent waiting for and undergoing review before merge. Most teams track only aggregate lead time, which hid nothing when writing dominated it, but AI collapses the writing portion toward zero and shifts the mass of lead time into review — so the aggregate stops telling you where the delay is. Once you split it, review latency becomes a first-class metric you can watch and act on. Useful sub-measures include time-to-first-review (how long a PR waits before anyone looks), review cycle time (how long from first review to merge), reviewer load (how many changes each reviewer is responsible for), and the ratio of code produced to code reviewed at depth. The last one exposes the core problem: if generation doubled but reviewing capacity did not, the excess is either sitting in a queue inflating lead time or being waved through shallowly and inflating change failure rate. The fix is partly measurement and partly investment — staffing review, providing reviewers with better code-context tooling so they can keep up, and keeping batch sizes small so each review is tractable. But you cannot manage the bottleneck you cannot see, so the measurement comes first: make review latency visible, then decide whether to add capacity or reduce the volume flowing into it.

Q: What does the 2025 DORA report mean by AI being an amplifier?

The 2025 DORA report central finding is that AI primary role is as an amplifier: it magnifies an organization existing strengths and weaknesses rather than uniformly improving or harming performance. Concretely, this means the returns on AI investment come not from the tools themselves but from the underlying sociotechnical system — the team practices, the review discipline, the testing rigor, the batching habits, and the organizational structure into which the AI is introduced. A team that already delivers well, with small batches, strong automated testing, and fast thorough review, gets faster when it adds AI, because the extra output flows through a system that can absorb it. A team that delivers poorly, with big-bang changes, weak tests, and overloaded reviewers, gets worse when it adds AI, because the tool amplifies the volume flowing through an already-broken gate. This is the single most important framing for leaders because it inverts the usual instinct. When delivery metrics regress after AI adoption, the intuitive response is to adopt more AI or buy a better coding assistant. The amplifier finding says that is precisely wrong: the leverage is in fixing the review, testing, and batching discipline that AI is amplifying. The tool matters far less than whether the delivery system can absorb what it produces, which is why a struggling team should invest in the system, not the tooling.

Q: Should we slow down AI adoption if it is hurting our delivery metrics?

No — the answer is not to slow AI adoption but to fix the system AI is exposing, because the productivity gains are real and the delivery regression is a symptom of a pre-existing weakness that AI made visible. Slowing adoption gives up the genuine individual-productivity benefit (three-quarters of developers report it) to avoid confronting the actual problem, which is that your review, testing, and batching discipline could not absorb the increased output. That weakness was there before AI; it simply did not bite because writing code was the constraint that kept volume low. The constructive response has three parts. First, restore small batch sizes deliberately — encourage or enforce smaller pull requests so each change stays reviewable and low-risk even though it is faster to produce. Second, invest in the review bottleneck — staff it, tool it, and measure its latency — because that is where the extra output now piles up. Third, harden the gates that contain blast radius, including feature flags so a bad change is a toggle rather than a rollback, and CI and eval gates sized for the new volume, plus scrutiny of AI-generated tests that may assert tautologically. Do these and the amplifier works for you: individual productivity and delivery performance rise together rather than diverging. The teams that win with AI are not the ones that adopted it fastest or slowest, but the ones whose delivery systems could absorb what it produced.

Q: Why do individual productivity and team delivery diverge with AI?

They diverge because AI accelerates one stage of software delivery — writing code — while leaving the other stages, especially review, testing, and integration, at their original speed, so the system becomes unbalanced. Individual productivity is felt at the writing stage: a developer using AI produces more code faster and genuinely experiences a large productivity gain, which is why survey after survey shows high satisfaction and perceived speed. Team delivery performance, however, is a property of the whole pipeline from commit to stable production, and a pipeline is only as fast and safe as its slowest, riskiest stage. When AI floods the writing stage with output, that output has to pass through review and testing that did not speed up, so it either queues (inflating lead time) or passes through with less scrutiny (inflating change failure rate). Either way, team delivery does not improve in proportion to the individual gains, and can regress. There is also the batch-size effect compounding it: the extra individual output tends to arrive as larger changesets, which are inherently riskier and slower to review, widening the gap further. This divergence is why measuring individual productivity as a proxy for delivery is now actively misleading, and why the discipline of the AI era is to measure the whole system — batch size, review latency, change failure rate — rather than celebrating the individual-output metrics that AI trivially inflates.

Satyam Kumar

返回博客

ai-architecture

AI Broke Your DORA Dashboard: What to Measure When Everyone Ships Faster

By Satyam KumarAugust 6, 20269 min read

dora metrics ai-assisted development software delivery devops change failure rate batch size engineering metrics developer productivity devex four keys ai coding tools code review engineering leadership delivery performance ci cd 2026

AI Broke Your DORA Dashboard: What to Measure When Everyone Ships Faster

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

AI Broke Your DORA Dashboard: What to Measure When Everyone Ships Faster

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment