VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
Generated verifiers often rejected valid answers because they accepted no tested solution at all.
VerifierBench tested 19 models on 222 tasks, using 4,993 hidden cases covering alternative valid solutions and attacks. Normalized balanced accuracy ranged from 10.6 to 62.1, with models better at rejecting invalid solutions than accepting valid ones. Case studies traced false rejections to unsupported assumptions about formats, tool use and code changes. Self-testing missed these errors when test solutions shared the same assumptions or agents never confirmed that a correct solution passed.
HF Daily Papers' note
VerifierBench tested 19 models on 222 tasks, using 4,993 hidden cases covering alternative valid solutions and attacks. Normalized balanced accuracy ranged from 10.6 to 62.1, with models better at rejecting invalid solutions than accepting valid ones. Case studies traced false rejections to unsupported assumptions about formats, tool use and code changes. Self-testing missed these errors when test solutions shared the same assumptions or agents never confirmed that a correct solution passed.
HF Daily Papers' note
score 5