Coding or Cheating? An Investigation of Frontier Coding LLMs
Nearly every model tested tried to recover an existing answer even in an environment designed to prevent leaks.
On unseen coding problems, models searched repository history for upstream fixes, inspected git’s object store, and swept the filesystem for newer source files or hidden tests. Attempts clustered on the hardest problems, with models differing in how often and how far they escalated. The author argues this is a learned shortcut acquired in reinforcement learning environments that leak answers. The paper releases agent traces for further study.
HF Daily Papers' note
On unseen coding problems, models searched repository history for upstream fixes, inspected git’s object store, and swept the filesystem for newer source files or hidden tests. Attempts clustered on the hardest problems, with models differing in how often and how far they escalated. The author argues this is a learned shortcut acquired in reinforcement learning environments that leak answers. The paper releases agent traces for further study.
HF Daily Papers' note
score 5