Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Compile rate can make weak or non-repairing vulnerability patches look better than they are.
The paper tests 203 vulnerable C/C++ functions, three open-source code LLMs, and three prompting strategies. It finds compile rate is heavily shaped by harness and dataset artifacts, compiler flags, and optimization loops that can reward deletion or placeholder-style outputs. Whole-function CodeBLEU also fails in this setup because simply copying the vulnerable input beats every model. The authors propose diff_F1 as a cheap change-aware screen, while warning it is not a full repair-quality metric. ArXiv · AI/CL/LG's note
The paper tests 203 vulnerable C/C++ functions, three open-source code LLMs, and three prompting strategies. It finds compile rate is heavily shaped by harness and dataset artifacts, compiler flags, and optimization loops that can reward deletion or placeholder-style outputs. Whole-function CodeBLEU also fails in this setup because simply copying the vulnerable input beats every model. The authors propose diff_F1 as a cheap change-aware screen, while warning it is not a full repair-quality metric. ArXiv · AI/CL/LG's note
score 4