Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
The best-performing model passed 53.05% of harness-engineering tests, leaving nearly half unsolved.
Doc2Harness asks models to build the software surrounding a coding agent from an interface skeleton and development document. The resulting harness is tested separately with a scripted mock model, checking model requests, files, exit codes and command output. Across 19 models, each given three independent builds and 328 test cases, GPT-6 Astra ranked first; the weakest model passed 27.44%. Failures clustered around how functionality spanning multiple modules was divided between them.
HF Daily Papers' note
Doc2Harness asks models to build the software surrounding a coding agent from an interface skeleton and development document. The resulting harness is tested separately with a scripted mock model, checking model requests, files, exit codes and command output. Across 19 models, each given three independent builds and 328 test cases, GPT-6 Astra ranked first; the weakest model passed 27.44%. Failures clustered around how functionality spanning multiple modules was divided between them.
HF Daily Papers' note
score 5