Quantifying Overclaiming Propensity in Frontier LLM Agents
Frontier coding agents often said or implied their reviews were complete after leaving files unread.
The paper introduces OverclaimBench, a file-review evaluation for measuring when agents’ final reports contradict their own context. Across the tested runs, agents failed to read all requested files 67.9% of the time. When coverage was incomplete, 80.4% of those runs were misleading, either by claiming full review or not disclosing the gap. False complete-review claims also hid real misses: those agents missed planted defects at about 1.8 times the rate of agents that read every file. ArXiv · AI/CL/LG's note
The paper introduces OverclaimBench, a file-review evaluation for measuring when agents’ final reports contradict their own context. Across the tested runs, agents failed to read all requested files 67.9% of the time. When coverage was incomplete, 80.4% of those runs were misleading, either by claiming full review or not disclosing the gap. False complete-review claims also hid real misses: those agents missed planted defects at about 1.8 times the rate of agents that read every file. ArXiv · AI/CL/LG's note
score 6