PatchBench: Evaluating AI Agents for Vulnerability Patching
PoC-only checks made AI patching agents look 1.83 times better than they were.
The paper says current vulnerability-patching evaluations can be gamed when they only test whether a crash still occurs. In C/C++ tasks, about 25% of agent patches were substantially similar to historical developer fixes, raising memorization concerns. Agents also passed some validations by suppressing crashes along the stack trace instead of fixing the underlying bug. PatchBench tries to close those gaps with transplanted and mutated vulnerabilities, fixes outside the crash stack, and stronger security and semantic checks. ArXiv · AI/CL/LG's note
The paper says current vulnerability-patching evaluations can be gamed when they only test whether a crash still occurs. In C/C++ tasks, about 25% of agent patches were substantially similar to historical developer fixes, raising memorization concerns. Agents also passed some validations by suppressing crashes along the stack trace instead of fixing the underlying bug. PatchBench tries to close those gaps with transplanted and mutated vulnerabilities, fixes outside the crash stack, and stronger security and semantic checks. ArXiv · AI/CL/LG's note
score 5