HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models
A small-model scanner recovered 65 of 95 public AI-found CVEs under blinded judging.
HoF-Bench pins real vulnerable commits from eight repositories and withholds CVE names, descriptions, fixes, and expected mechanisms from analyzers. The paper says credit was given only when a blinded judge found the same code path, root cause, attack condition, and impact. The tested detector backbones were open-weight and small or “flash”-tier proprietary models, with no frontier model doing detection. Misses clustered in C infrastructure code. ArXiv · AI/CL/LG's note
HoF-Bench pins real vulnerable commits from eight repositories and withholds CVE names, descriptions, fixes, and expected mechanisms from analyzers. The paper says credit was given only when a blinded judge found the same code path, root cause, attack condition, and impact. The tested detector backbones were open-weight and small or “flash”-tier proprietary models, with no frontier model doing detection. Misses clustered in C infrastructure code. ArXiv · AI/CL/LG's note
score 6