ExplainBench: Evaluating Code Explanations from Agents
ExplainBench tests whether coding agents’ explanations actually help another model understand the bug and the patch.
The benchmark asks questions about the buggy code’s intended behavior and the effect of the agent’s patch. The authors report that it ranks agents differently from SWE-bench Verified, treating explanation quality as its own measure. They also find that agent explanations often say a patch is correct when it is not. An added audit agent, using extra tests to validate and refine explanations, improved results for every evaluated agent. HF Daily Papers' note
The benchmark asks questions about the buggy code’s intended behavior and the effect of the agent’s patch. The authors report that it ranks agents differently from SWE-bench Verified, treating explanation quality as its own measure. They also find that agent explanations often say a patch is correct when it is not. An added audit agent, using extra tests to validate and refine explanations, improved results for every evaluated agent. HF Daily Papers' note
score 5