Harness-Delta Attribution: Dissecting Gains From Harness Evolution
Overfitting and extra inference-time compute explained an average 68% of gains across 21 settings with significant search-set improvements.
The study tested 11 executor models and three frameworks that optimize prompts, tools, skills and orchestration around fixed model weights. Its attribution protocol separates benchmark-artifact overfitting, gains explained by test-time scaling and a remainder that only upper-bounds generalizable improvement. Overfitting and scaling together explained more than half the gain in 14 of those 21 settings. Search-set gains often failed to transfer to held-out tasks, and validation-based signals were insufficient to prevent overfitting.
HF Daily Papers' note
The study tested 11 executor models and three frameworks that optimize prompts, tools, skills and orchestration around fixed model weights. Its attribution protocol separates benchmark-artifact overfitting, gains explained by test-time scaling and a remainder that only upper-bounds generalizable improvement. Overfitting and scaling together explained more than half the gain in 14 of those 21 settings. Search-set gains often failed to transfer to held-out tasks, and validation-based signals were insufficient to prevent overfitting.
HF Daily Papers' note
score 5