TL;DR
Does a 10-point jump on a coding leaderboard actually help you ship more code? According to this study, the answer is a resounding "not really." By analyzing a "natural experiment" where a top-tier AI model was suddenly banned, researchers found that switching to significantly lower-scoring models had no measurable impact on developer productivity or pull request merge rates.
What Problem Does This Paper Address?
AI companies frequently use benchmarks like SWE-bench Pro to claim their latest models are superior. However, there is a massive gap between solving a isolated, pre-defined bug in a benchmark and doing "real work." Real-world software engineering involves navigating vague requirements, adhering to project-specific styles, and—most importantly—getting code reviewed and merged by humans.
Until now, we haven't known if these benchmark gains translate into developers actually getting more work done. Measuring this is notoriously difficult because of "confounders": maybe the people who use the newest models are just better developers to begin with, or maybe they only use the "strongest" model for the hardest tasks, making that model look less productive by comparison.
Methods
The author, Yining Hua, took advantage of a unique natural experiment. On June 12, 2026, a U.S. export control directive abruptly banned access to Claude Fable 5 for 19 days. This forced a specific group of power users to switch to Opus 4.8, a model that scored 11 points lower on SWE-bench Pro.
To isolate the effect of the model itself, the study used a Difference-in-Differences (DiD) design:
- Treated Group: 297 developers who were using Fable 5 right before the ban.
- Control Group: 8,908 developers who stayed on Opus 4.8 the entire time.
- Fixed Effects: The model controlled for individual developer habits and specific calendar dates to ensure the results weren't skewed by a "productive" person or a "slow" holiday week.
The study also looked at five other major model upgrades throughout 2026 (involving 1,208 developers and over 15,000 pull requests) to see if "leveling up" a model improved output.
Datasets
The researchers used the GitHub Search API to collect public Pull Requests (PRs) created with Claude Code.
- Identification: Claude Code automatically adds a "Co-Authored-By" line to commits, allowing the researchers to identify exactly which model (e.g., Fable 5 vs. Opus 4.8) was used for every single PR.
- Scope: They tracked PRs created between November 2025 and October 2026.
- Validation: They manually checked a sample of 2,019 PRs to see who merged them and whether they were approved by others.
Key Insights and Ideas
- The "Good Enough" Ceiling: The most likely reason for the flat results is that the tasks developers currently give to AI are already within the "capabilities" of older models. A survey of Anthropic engineers found they mostly use AI for tasks that are "easy to check and cheap to get wrong." If you're only asking the AI to write a unit test or a simple refactor, a "smarter" model doesn't make that task "more merged."
- The Human Bottleneck: The study found that 95% of PRs were merged by the authors themselves, usually in under 30 minutes. This suggests that for many AI-assisted tasks, the "review" process is almost non-existent or extremely fast. If the human is the one clicking "merge" on their own AI-generated code, a slightly better model doesn't change the social or technical threshold for what gets shipped.
- Benchmark Misalignment: Benchmarks like SWE-bench Pro measure if a model can fix a bug given a specific test case. They don't measure the "soft" parts of engineering: clarifying a vague request, architectural planning, or responding to a human reviewer's feedback.
Key Results
Despite the marketing hype around benchmark scores, the quantitative impact on real work was negligible:
- Merge Rates: Having access to Fable 5 (the "stronger" model) increased the 7-day merge rate by only 1.4 percentage points, a result that was not statistically significant.
- Model Upgrades: Across five different model upgrades, the merge rate actually saw a slight (but non-significant) decrease of 0.9 percentage points.
- Volume of Work: There was no significant increase in the number of PRs merged per week (+2.4%).
- Quality: Switching to a higher-scoring model did not significantly reduce the rate of failed automated checks or tests.
Limitations and Future Work
- The "Merge Rate" Proxy: While merge rates are a standard metric for productivity, they don't capture "time saved." A developer might use a better model to finish a task in 10 minutes instead of 20, but if they spend the remaining 10 minutes in a meeting, their "PRs per week" won't change.
- Sample Bias: The study only tracked developers who used the default "Co-Authored-By" attribution. These might be a specific subset of developers (e.g., open-source contributors) whose habits differ from those in private enterprise.
- Task Complexity: The study couldn't perfectly measure if developers started giving the "better" models significantly harder tasks, though PR description lengths and file counts suggested the work remained similar.
Future research should focus on randomized controlled trials that can track active coding time and the complexity of tasks handed off to AI, rather than just the final binary outcome of a merged PR.
Summary updated