Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
Task-CoEvolve claims full-set harness search performance with 80% fewer optimization evaluations.
The method changes which validation tasks are run as the harness is rewritten, favoring tasks where candidate harnesses disagree. It uses variance-weighted sampling from past outcomes, then estimates full-set scores from the sampled subset by accounting for sampling probabilities. The paper reports results on online text classification and Terminal-Bench 2.1, where it beat subset baselines and matched full-set search’s final performance. Code is promised for release. HF Daily Papers' note
The method changes which validation tasks are run as the harness is rewritten, favoring tasks where candidate harnesses disagree. It uses variance-weighted sampling from past outcomes, then estimates full-set scores from the sampled subset by accounting for sampling probabilities. The paper reports results on online text classification and Terminal-Bench 2.1, where it beat subset baselines and matched full-set search’s final performance. Code is promised for release. HF Daily Papers' note
score 4