METR Didn't Reverse Its 19% AI Slowdown (2026 Update)

METR's early-2025 study found that experienced developers were 19% slower with AI, the opposite of what the developers themselves predicted. A randomized controlled trial is a study design that randomly assigns each unit of work to a treatment or control condition so that differences in outcome can be attributed to the treatment rather than to who did what. Here the units were coding tasks, and the treatment was permission to use AI tools.

The sample was small but deliberately expert. According to METR (2025), the study recruited 16 experienced developers who completed 246 total tasks, all real issues in mature repositories the participants had contributed to for multiple years, averaging 22,000+ stars and more than a million lines of code. The tooling was frontier for its moment: primarily Cursor Pro with Claude 3.5 and 3.7 Sonnet. This matters because the result is often quoted as "AI makes developers slower" full stop, when the actual finding is narrower and more interesting: on large, familiar, high-context codebases, expert developers were slower with the AI tools available in early 2025.

That distinction is the whole reason the number is easy to abuse. The population most likely to be slowed down, senior maintainers deeply fluent in a codebase, is exactly the population whose time is most expensive, so the finding felt important. But it says almost nothing about a junior developer on an unfamiliar service, or about greenfield work where there is no existing context to hold in your head. We unpack that context-dependence at length in our analysis of the AI coding productivity paradox, which is the companion piece to this one.

The most durable lesson from the 2025 study is not the slowdown itself but the perception gap around it. The perception gap is the distance between how much developers believe AI is helping and what controlled measurement shows it is doing.

METR captured three numbers on the same people doing the same tasks. Before starting, developers expected AI to speed them up by 24%. After finishing, they still believed AI had sped them up by 20%. The measurement said they were 19% slower. According to METR (2025), that is a roughly 39-percentage-point gap between post-hoc belief (+20%) and measured reality (-19%), and the belief did not budge even after the work was done.

This is the finding with the longest shelf life, because it is not about any specific model version. It is about how badly self-report tracks reality for this kind of work. The broader industry data makes the same point from the other side. According to Google's DORA 2025 report (2025), which surveyed nearly 5,000 technology professionals and was published on September 24, 2025, over 80% of respondents said AI had enhanced their productivity, and developers now spend a median of two hours daily working with AI. Both things can be true at once: adoption and satisfaction are high, and controlled measurement of completion time on expert tasks pointed the other way. If your rollout decision rests on a satisfaction survey, DORA is what you will see. METR is what you would measure. We pull apart that same tension between felt speed and shipped outcomes in our breakdown of the DORA 2025 acceleration-versus-incidents data.

No. METR did not reverse the finding, and it did not publish a speedup. Its February 24, 2026 update kept the same sign at the point estimate and changed only how much METR trusts the number.

Here is the sign convention that trips up almost every secondary write-up. METR reports results as a signed "speedup," where a negative speedup is a slowdown. According to METR (2026), the returning developers, a subset of the original cohort, now show "a speedup of -18% with a confidence interval between -38% and +9%." A speedup of -18% is an 18% slowdown. The newly recruited developers came in at -4%, with a confidence interval between -15% and +9%, also a slight slowdown at the point estimate. The 2026 post is titled "We are Changing our Developer Productivity Experiment Design," which is not the title you give an announcement that your headline result flipped.

So the "37-point swing" and "18% speedup" you may have seen are artifacts of reading -18% as +18%. The point estimate for returning developers moved about one point in a year. What METR does say, carefully, is directional and qualitative, not a measured speedup: it believes it is "likely that developers are more sped up from AI tools now, in early 2026, compared to our estimates from early 2025," while adding that "because of the selection effects in our experiment, our data is only very weak evidence for the size of this increase." That is a hypothesis with a caveat attached, not a result.

METR labeled its own 2026 estimate unreliable because a large fraction of developers refused to do the no-AI half of the experiment. That single behavior is enough to break the comparison, and METR said so plainly.

According to METR (2026), "30% to 50% of developers told us that they were choosing not to submit some tasks" because they did not want to do them without AI. When people opt out of the harder, unaided condition, the tasks that remain in that arm are not a fair comparison group, so the measured gap understates the true effect. METR's own conclusion is that "our estimate reported above is a lower-bound on the true productivity effects of AI," and that "the data from our new experiment gives us an unreliable signal." A lower bound is a floor: the real effect could be more favorable to AI, but the experiment as run cannot tell you by how much.

Two more confounds stacked on top of that. Pay was cut from $150/hr to $50/hr between rounds, which introduces a new selection effect in who agreed to participate at all. And METR notes that its "measurements of time-spent on each task are unreliable for the fraction of developers who use multiple AI agents concurrently," because running several agents at once makes wall-clock-per-task ambiguous. Faced with all three problems, METR did the responsible thing: it did not ship a corrected number. It announced a redesign, with options including randomizing at the developer level instead of the task level, or reverting to a fixed-task design. As of the February 2026 post, no new reliable productivity figure exists to replace the original -19%.

The reason METR's 2026 number collapsed is selection bias, and it is the single most important thing to internalize before you try to measure this yourself. Selection bias is the distortion that occurs when the units that end up in each condition differ systematically for reasons related to the outcome.

Think about what happens mechanically. If developers get to skip tasks they do not want to do without AI, the no-AI arm quietly loses exactly the tasks where working without AI would have been most painful and slowest. The unaided condition ends up stocked with the easier residue. Compared against that softened baseline, the AI-allowed arm looks relatively worse off than it should, which pushes the measured speedup toward zero or negative from below. That is why METR frames -18% as a floor rather than a point estimate you can trust.

This is not an exotic statistical footnote. It is the failure mode that ruins most in-house "does AI help us" experiments, which almost never enforce completion of the control arm. It also compounds with a second measurement trap: counting a fast first draft as a win while ignoring what happens downstream. A change that ships quickly but has to be rewritten twice is not faster in any sense that matters, and generated code has a documented tendency to churn. Our data-driven look at AI code churn and cloning is the reason we insist on tracking rework alongside raw completion time in any uplift study.

You cannot lift the -19% or the -18% and apply it to your organization, because both were measured under conditions that are almost certainly not yours. The single most reliable takeaway from METR is that the effect is highly context-dependent, which is precisely what makes any specific number non-transferable.

Run down the variables that move the result. The 2025 study used senior maintainers on codebases they had lived in for years, where the human already holds enormous context that the model has to be laboriously fed. Your task mix, if it includes unfamiliar services, boilerplate, tests, and glue code, sits in a different part of the distribution. The tooling was Cursor Pro with Claude 3.5 and 3.7 Sonnet in early 2025; model and harness quality have moved since, and the harness often matters more than the raw model, which is the whole thesis of our piece on how agent scaffolding beats model upgrades on SWE-Bench. Even the choice of tool changes the ergonomics enough to matter, as we lay out in the Cursor vs Claude Code 2026 guide. Borrowing METR's number assumes all of that is constant. It is not.

The correct use of METR is not as a benchmark to copy. It is as a warning that self-report will lie to you and that the effect depends on conditions you can only characterize by measuring your own.

Measure uplift with a task-level randomized comparison, not a survey, and enforce completion of both arms so you do not reproduce METR's selection trap. This is the lightweight in-house method that survives contact with the objections above.

The design is simple enough to run in a two-week sprint:

The output is not a universal truth about AI and productivity. It is a decision-grade number for your team, your stack, and your codebase, which is the only kind of number worth acting on. Across the engineering organizations we advise at Particula Tech, the teams that measure this way almost always find a more nuanced picture than either the doom headline or the vendor deck: large gains on some task classes, real slowdowns on others, and a rollout decision that should be made per workflow rather than in one blanket motion.

Trust the direction of METR's original 2025 finding as evidence that expert developers on familiar code can be slowed by AI, and trust the perception gap as a durable warning that self-report runs optimistic by tens of points. Skip the "METR reversed to an 18% speedup" narrative entirely: it is a misread of a negative sign, METR published no speedup, and METR itself calls its 2026 estimate an unreliable lower bound while it redesigns the study.

The practical recommendation is blunt. Do not run your AI tooling strategy on either METR number or on a satisfaction survey. Run it on a task-level A/B you conduct yourself, with the control arm enforced and rework tracked alongside speed. That is a two-to-four-week exercise, and it will tell you more than any borrowed benchmark. If you want that measurement designed and run rigorously before you commit budget to a fleet-wide rollout, Particula Tech's AI tooling audits build exactly that instrumentation, and the broader trade-offs live in our AI development tools pillar guide. Measure first, then scale the tools that actually move your numbers.