HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
The benchmark tests whether LLMs can build and improve the execution harness around an agent, not just solve tasks inside one.
HarnessDev has agents create a runnable system from a minimal seed, then revise it using downstream feedback. The paper reports results across six creator models, four domains, and 2,207 downstream instances. Generated harnesses lagged human-built references on code, search, and research, but matched or beat selected references on writing and machine-learning experimentation. Evolution helped sometimes, but gains were unstable and only partly carried to held-out tasks. HF Daily Papers' note
HarnessDev has agents create a runnable system from a minimal seed, then revise it using downstream feedback. The paper reports results across six creator models, four domains, and 2,207 downstream instances. Generated harnesses lagged human-built references on code, search, and research, but matched or beat selected references on writing and machine-learning experimentation. Evolution helped sometimes, but gains were unstable and only partly carried to held-out tasks. HF Daily Papers' note
score 5