EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?
The best agent-built evaluators matched target benchmarks on about 75% of model-pair rankings, below the 91% ceiling set by disagreements among the targets.
EvalResearchBench tested nine researcher agents building evaluators for 13 candidate models under fixed time and API budgets. Agents selected or created tasks, implemented graders, and revised them through pilot tests. The strongest evaluator on development targets did not lead on hidden targets, while a human-designed sample of public tasks remained a strong baseline. Pilot feedback helped agents repair problems, but their evaluators could still truncate answers, exhaust budgets, or let a few questions dominate a domain score.
HF Daily Papers' note
EvalResearchBench tested nine researcher agents building evaluators for 13 candidate models under fixed time and API budgets. Agents selected or created tasks, implemented graders, and revised them through pilot tests. The strongest evaluator on development targets did not lead on hidden targets, while a human-designed sample of public tasks remained a strong baseline. Pilot feedback helped agents repair problems, but their evaluators could still truncate answers, exhaust budgets, or let a few questions dominate a domain score.
HF Daily Papers' note
score 5