Higher-Scoring Coding Models Did Not Get More Work Done
Access to Fable 5 raised the estimated seven-day pull-request merge rate by 1.4 percentage points, a statistically insignificant result.
The study tracked 297 Fable 5 users through a 19-day suspension that pushed most onto Opus 4.8, which scored 11 points lower on SWE-Bench Pro. It compared them with 8,908 developers who remained on Opus 4.8. The estimated merge-rate effect ranged from −1.4 to +4.3 percentage points, leaving both a decline and an improvement plausible. Comparisons across five other model upgrades also found no statistically significant gains, nor did the study find a significant increase in pull requests merged per week.
ArXiv · AI/CL/LG's note
The study tracked 297 Fable 5 users through a 19-day suspension that pushed most onto Opus 4.8, which scored 11 points lower on SWE-Bench Pro. It compared them with 8,908 developers who remained on Opus 4.8. The estimated merge-rate effect ranged from −1.4 to +4.3 percentage points, leaving both a decline and an improvement plausible. Comparisons across five other model upgrades also found no statistically significant gains, nor did the study find a significant increase in pull requests merged per week.
ArXiv · AI/CL/LG's note
score 5