From Tool Spend To Team Value: Measuring AI Coding

🔍 Read the full analysis: From Tool Spend To Team Value: Measuring AI Coding on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get hardware and tech essentials delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Engineering teams are reporting AI coding budgets from $150 to about $3,000 per developer each month, but a report drawing on interviews at 27 workplaces finds that many companies cannot yet say what measurable business value that spending produced. The findings arrive as organizations set 2027 budgets and some discounted subscriptions are due to end, sharpening the need to distinguish useful work from higher tool consumption.

Gergely Orosz published the report in The Pragmatic Engineer on October 20, based on conversations with engineering leaders and senior engineers at 27 workplaces, alongside aggregate data from AI code-review vendors CodeRabbit and Gitar. The source article says its figures are drawn from the publication’s freely readable material. Spending varied sharply: one publicly traded company with 1,500 engineers capped individual budgets at $150 a month, while two-person European agency Ix Technologies spent about $3,000 per developer. Company size alone did not explain the difference.

Reported output also varied. CodeRabbit said developers on its platform created 113% more pull requests per week than a year earlier, a vendor measure from teams using its review product. A Microsoft study of tens of thousands of engineers during an early-2026 rollout found adopters merged about 24% more pull requests than they otherwise would have over four months. Microsoft’s researchers cautioned that a merged pull request does not by itself establish the value delivered. At some surveyed companies, pull-request counts were flat; in one case, PRs grew to three times their former size.

Evidence from controlled studies is less conclusive. METR’s early-2025 study found experienced open-source developers took 19% longer with AI. A February follow-up estimated an 18% speedup for returning developers and 4% for new developers, but both estimates had confidence intervals that included zero. METR also described the follow-up data as unreliable, citing task selection by participants. These different measures are not directly interchangeable: they cover different workers, tasks, time periods and definitions of productivity.

Budgets Need Delivery Measures

The immediate issue for engineering leaders is that tool bills are easier to count than value. The source recommends tracking cost per accepted change, rework, change-failure rates and whether teams close more user stories than arrive. Those measures can connect spending to work that passes review and reaches users, unlike raw token use or PR volume on its own.

The distinction matters as discounts end and usage-based billing exposes teams more directly to model costs. The source reports that several companies expect discounted subscriptions to expire, with some saying costs could rise five- to tenfold if they do not optimize. That is an attributed forecast from those companies, not a universal estimate. Without agreed outcome measures, budget decisions risk rewarding greater consumption or larger code submissions rather than reliable delivery.

The reported use cases are specific rather than universal: issue triage, understanding unfamiliar code, prototypes and backlog work on simple, consistent technology stacks. A SaaS company with about 600 engineers reported little movement in merged-PR velocity, while a staff engineer described customer issue triage shrinking from much of a day to minutes of agent work plus up to an hour of checking. The example suggests that benefits may appear in time saved on individual tasks even when broad team output metrics barely change.

Why AI Coding Bills Are Changing

The report describes spending on API plans as having peaked in June and July and then roughly halved for some companies, which Orosz attributed to the release of Opus 5.5 and a move to open models. That account is not a market-wide spending dataset. It also notes that spending caps are common and, according to the people interviewed, can be circumvented.

Billing policy is changing alongside adoption. The source says GitHub moved all Copilot plans to usage-based billing on June 1, pricing tokens at each model’s API rate and removing a fallback to a cheaper model. GitHub explained: “Agentic usage is becoming the default, and it brings significantly higher compute and inference demands.” This helps explain why companies are revisiting budgets, but does not establish that every customer’s costs will rise by the same amount.

Some teams describe clearer returns where systems are relatively uniform. Ix Technologies, which builds with C# and Blazor, said it ships about 60 pull requests per developer each week while clearing a backlog dating back roughly ten years. Cofounder Rogier van der Hee said the two-person company felt as though it could do the work of five to seven people before AI. That is his assessment, not an independently verified productivity comparison. The source also cites one multinational team that closed three times as many user stories per month and, for the first time, more than arrived.

Productivity Evidence Remains Mixed

The figures do not establish a single productivity effect across engineering teams. Vendor platform data, company-level comparisons, controlled studies and employee self-reports use different methods and populations. METR said its February follow-up results were unreliable because 30% to 50% of developers held back tasks they did not want to do without AI. Its survey of 349 technical workers found a median self-reported 1.4-to-2-times change in work value, but METR also said participants in its 2025 study overestimated AI’s effect by 40 percentage points.

It is also unclear how broadly the reported gains translate into shipped, dependable software, or how much they reduce total labor and operating costs. The source cites a Veracode test in which 44% of AI coding tasks produced a known vulnerability, but provides no further test details in the material supplied here. The result should not be read as a rate for all AI-generated code or all deployed software. The article’s claim that some companies could face five- to tenfold cost increases after discounts end is likewise based on company accounts, with no common baseline or confirmed outcome.

Track Results as Discounts Expire

Companies setting 2027 budgets will need to compare costs with measures that reflect accepted, maintainable work. The source points to cost per accepted change, rework, change-failure rates and stories closed versus stories arriving as useful indicators. It does not provide a shared measurement standard or evidence that companies have adopted one.

As discounted subscriptions expire and usage-based plans expose model costs, teams may reassess caps, models and the tasks assigned to coding agents. Whether that produces lower bills or better delivery remains to be seen. The next useful evidence will be company-level reporting that ties AI costs to code quality, maintenance and delivered work over a defined period—not simply a count of generated code or pull requests.

Key Questions

How much are companies spending on AI coding tools?

The workplaces surveyed reported budgets ranging from $150 to about $3,000 per developer per month. The figures are examples from 27 workplaces, not a representative estimate for all companies.

Does the report show that AI coding increases productivity?

It reports gains in some settings, including more merged pull requests at Microsoft and more weekly pull requests on CodeRabbit’s platform. Other companies reported flat PR counts, and METR’s controlled-study estimates were uncertain. The evidence does not establish one productivity effect for every team.

What should teams measure besides AI tool usage?

The source recommends tracking cost per accepted change, rework, change-failure rate and whether teams close more user stories than arrive. These measures aim to connect spending to reviewed work and delivery outcomes.

Where did teams report the clearest benefits?

Examples included customer issue triage, tracing why code was written a certain way, building prototypes and clearing backlogs on simple, consistent technology stacks. The examples are company accounts and do not prove the same results will occur elsewhere.

Are AI-generated changes safe to merge?

Not automatically. The source cites a Veracode test in which 44% of AI coding tasks produced a known vulnerability. It does not provide enough detail to generalize that rate to all AI-generated code, so teams still need security checks and code review.

Source: ThorstenMeyerAI.com