A good year ago, in my post on vibe coding, I played through thesis, antithesis and synthesis: democratization of software on one side, a technical time bomb on the other, and in between the developer as architect and auditor. Since then, something has arrived that was missing at the time: solid data. Time for a balance sheet of the three big promises.
Promise 1: "With AI, everyone is faster"
The most prominent counterexample came from the non-profit METR with a randomized controlled trial (July 2025): 16 experienced open-source developers worked through 246 real tasks in their own mature projects, sometimes with AI tools (mainly Cursor with Claude models) and sometimes without. The result was a minor sensation: with AI, they took 19 percent longer. Even more remarkable was the perception gap: the same developers estimated afterwards that AI had sped them up by 20 percent. Economists and ML experts had likewise forecast hefty speedups.
To be fair, context matters: the study looked at experienced developers in codebases they knew inside out. That is the scenario where AI can contribute least and review overhead hits hardest. The tools were the state of early 2025, and METR itself is evolving the experiment design. Still, the finding stays useful as a thorn in the side: the feeling of productivity is not evidence of productivity.
Promise 2: "Developers love the tools"
Half true. The Stack Overflow Developer Survey 2025 shows a split picture: 84 percent of respondents use AI tools or plan to, which is a record. At the same time, 46 percent distrust the accuracy of the results, with only 33 percent trusting it; overall positive sentiment toward AI tools fell from above 70 to 60 percent. The most-cited frustration (45 percent) nails the core problem: AI solutions that are "almost right, but not quite", where the debugging then takes longer than writing it yourself.
Adoption up, trust down looks like a contradiction, but I read it as professionalization. The tools have become everyday equipment and are judged with the sobriety that goes with it.
Promise 3: "The code gets better"
Here the data is most uncomfortable. GitClear analyzed 211 million changed lines of code from 2020 through 2024: the share of copy-pasted (duplicated) lines rose from 8.3 to 12.3 percent, while refactoring (restructuring and cleaning up existing code) collapsed from 25 percent of changes to under 10 percent. In 2024, copy-paste exceeded the number of moved lines for the first time. In short: more code is being produced and less is being maintained. That is precisely the "technical time bomb" from my earlier post, now with measurements.
And yet: the counter-calculation
Turning this into "AI coding has failed" reads the data as selectively as the hype faction did a year earlier. My own balance is on record in the vibe coding post: there are apps of mine in the App Store that simply would not exist without AI assistance. The studies measure experienced developers in familiar, mature codebases. They don't measure the weekend prototype, the boilerplate, the entry into an unfamiliar domain, or the one-person project where nobody has to wait for a colleague. That's where the gains are, and they are real.
Also real, however, is the flip side in market form: a flood of generated apps, hardly any of which find users. I dissected that here. Easier to build does not mean easier to last.
Balance: the synthesis has held
A year later, I feel confirmed in the earlier synthesis, and more precisely than I'd like. The bottleneck is no longer writing code but the judgment about it: spotting "almost right" before it reaches production. The METR study shows that this review work doesn't feel like it slows you down, even when it sometimes does. The Stack Overflow survey shows developers now know it. And GitClear shows what happens when the review work is skipped. The developer as auditor is no longer a forecast. It's the job description.
Sources
- METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, randomized trial with 16 developers and 246 tasks (paper on arXiv)
- METR: We are Changing our Developer Productivity Experiment Design, evolution of the experiment design
- Stack Overflow: Developer Survey 2025 - AI, usage and trust figures
- Stack Overflow Blog: Developers remain willing but reluctant to use AI, interpretation of the survey results
- GitClear: AI Copilot Code Quality: 2025 Data Suggests 4x Growth in Code Clones, analysis of 211 million changed lines of code 2020-2024