Higher-Scoring Coding Models Did Not Get More Work Done

Yining HuaarXiv 2026 · arXiv

TL;DR

Does a 10-point jump on a coding leaderboard actually help you ship more code? According to this study, the answer is a resounding "not really." By analyzing a "natural experiment" where a top-tier AI model was suddenly banned, researchers found that switching to significantly lower-scoring models had no measurable impact on developer productivity or pull request merge rates.

What Problem Does This Paper Address?

AI companies frequently use benchmarks like SWE-bench Pro to claim their latest models are superior. However, there is a massive gap between solving a isolated, pre-defined bug in a benchmark and doing "real work." Real-world software engineering involves navigating vague requirements, adhering to project-specific styles, and—most importantly—getting code reviewed and merged by humans.

Until now, we haven't known if these benchmark gains translate into developers actually getting more work done. Measuring this is notoriously difficult because of "confounders": maybe the people who use the newest models are just better developers to begin with, or maybe they only use the "strongest" model for the hardest tasks, making that model look less productive by comparison.

Methods

The author, Yining Hua, took advantage of a unique natural experiment. On June 12, 2026, a U.S. export control directive abruptly banned access to Claude Fable 5 for 19 days. This forced a specific group of power users to switch to Opus 4.8, a model that scored 11 points lower on SWE-bench Pro.

To isolate the effect of the model itself, the study used a Difference-in-Differences (DiD) design:

  1. Treated Group: 297 developers who were using Fable 5 right before the ban.
  2. Control Group: 8,908 developers who stayed on Opus 4.8 the entire time.
  3. Fixed Effects: The model controlled for individual developer habits and specific calendar dates to ensure the results weren't skewed by a "productive" person or a "slow" holiday week.

The study also looked at five other major model upgrades throughout 2026 (involving 1,208 developers and over 15,000 pull requests) to see if "leveling up" a model improved output.

The study design utilized the sudden suspension of Fable 5 as a natural experiment to compare productivity phases.

Datasets

The researchers used the GitHub Search API to collect public Pull Requests (PRs) created with Claude Code.

  • Identification: Claude Code automatically adds a "Co-Authored-By" line to commits, allowing the researchers to identify exactly which model (e.g., Fable 5 vs. Opus 4.8) was used for every single PR.
  • Scope: They tracked PRs created between November 2025 and October 2026.
  • Validation: They manually checked a sample of 2,019 PRs to see who merged them and whether they were approved by others.

Key Insights and Ideas

  • The "Good Enough" Ceiling: The most likely reason for the flat results is that the tasks developers currently give to AI are already within the "capabilities" of older models. A survey of Anthropic engineers found they mostly use AI for tasks that are "easy to check and cheap to get wrong." If you're only asking the AI to write a unit test or a simple refactor, a "smarter" model doesn't make that task "more merged."
  • The Human Bottleneck: The study found that 95% of PRs were merged by the authors themselves, usually in under 30 minutes. This suggests that for many AI-assisted tasks, the "review" process is almost non-existent or extremely fast. If the human is the one clicking "merge" on their own AI-generated code, a slightly better model doesn't change the social or technical threshold for what gets shipped.
  • Benchmark Misalignment: Benchmarks like SWE-bench Pro measure if a model can fix a bug given a specific test case. They don't measure the "soft" parts of engineering: clarifying a vague request, architectural planning, or responding to a human reviewer's feedback.

Key Results

Despite the marketing hype around benchmark scores, the quantitative impact on real work was negligible:

  • Merge Rates: Having access to Fable 5 (the "stronger" model) increased the 7-day merge rate by only 1.4 percentage points, a result that was not statistically significant.
  • Model Upgrades: Across five different model upgrades, the merge rate actually saw a slight (but non-significant) decrease of 0.9 percentage points.
  • Volume of Work: There was no significant increase in the number of PRs merged per week (+2.4%).
  • Quality: Switching to a higher-scoring model did not significantly reduce the rate of failed automated checks or tests.
Figure 3. Benchmark gains of 5 to 18 points came with no rise in merging. a, SWE-Bench Pro gain of the newer model over the older one, from Anthropic’s reported scores. b, Change in the same developers’ 7-day merge rate, with 95% intervals. For each upgrade, we used the PRs that developers who switched opened in the 7 days before and after the release, in linear probability models with developer fixed effects. The five upgrades in the main analysis are pooled by inverse-variance weighting (Appendix A). The light bar and open circle show Sonnet 4.6 to 5, which had too few developers who switched to be pooled. The suspension row is the reversal contrast, with developer and calendar-day fixed effects. Intervals are clustered by developer. The two February upgrades are not shown, because SWE-Bench Pro scores are not available for both models. All eight upgrades are in Table S3.

Limitations and Future Work

  • The "Merge Rate" Proxy: While merge rates are a standard metric for productivity, they don't capture "time saved." A developer might use a better model to finish a task in 10 minutes instead of 20, but if they spend the remaining 10 minutes in a meeting, their "PRs per week" won't change.
  • Sample Bias: The study only tracked developers who used the default "Co-Authored-By" attribution. These might be a specific subset of developers (e.g., open-source contributors) whose habits differ from those in private enterprise.
  • Task Complexity: The study couldn't perfectly measure if developers started giving the "better" models significantly harder tasks, though PR description lengths and file counts suggested the work remained similar.

Future research should focus on randomized controlled trials that can track active coding time and the complexity of tasks handed off to AI, rather than just the final binary outcome of a merged PR.

Summary updated

Abstract

New coding models advertise themselves on score improvements of coding benchmarks. Yet no study has tested if a model that scores higher on these benchmarks gets more real work done. On June 12, 2026, a U.S. export control directive banned access to Claude Fable 5 for 19 days, which forced users to switch to other models. During this time, most former Fable 5 users switched to Opus 4.8 (11 points lower on SWE-Bench Pro compared to Fable 5). We treat this suspension as a natural experiment, where the people and projects stayed the same and only the model changed, so we could isolate the effect of Fable 5 on user productivity. We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group. We also repeated the comparison across five model upgrades before and after the suspension in 2026 (1,208 developers, 15,442 pull requests). Fitting a difference-in-differences model with developer and date fixed effects, we found that having access to Fable 5 increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant. Similarly, no significant increases of merge rates were found with the upgrades (weighted mean: -0.9 percentage points [-1.9; +0.2]); all five estimates were negative (95% intervals included 0, did not differ from each other). We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).
BibTeX
@misc{hua2026higher,
  author = {Yining Hua},
  title = {{Higher-Scoring Coding Models Did Not Get More Work Done}},
  year = {2026},
  eprint = {2611.23216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23216},
}
Figure