Megadose The AI curve, compressed.

Higher-Scoring Coding Models Did Not Get More Work Done

· ArXiv · AI/CL/LG ·
Access to Fable 5 raised the estimated seven-day pull-request merge rate by 1.4 percentage points, a statistically insignificant result.

The study tracked 297 Fable 5 users through a 19-day suspension that pushed most onto Opus 4.8, which scored 11 points lower on SWE-Bench Pro. It compared them with 8,908 developers who remained on Opus 4.8. The estimated merge-rate effect ranged from −1.4 to +4.3 percentage points, leaving both a decline and an improvement plausible. Comparisons across five other model upgrades also found no statistically significant gains, nor did the study find a significant increase in pull requests merged per week.

ArXiv · AI/CL/LG's note

score 5

Categories: Research