Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
The paper proposes an audit protocol that tests whether an AI research agent’s result was recoverable without the claimed discovery path.
DCP turns research claims into executable recovery and feedback tests, with sealed evaluations and matched agents given only registered starting information plus observed web content. Its Core standard requires controls, zero observed recoveries, and a finite-sample recovery bound in a fresh registered episode. In two audits, on SQLite optimization and virtual catalyst control, the authors report zero recoveries across 96 episodes and an upper bound of 0.0468. Optional feedback tests found truthful feedback recovered targets while neutral feedback did not, under the paper’s registered null calibration. HF Daily Papers' note
DCP turns research claims into executable recovery and feedback tests, with sealed evaluations and matched agents given only registered starting information plus observed web content. Its Core standard requires controls, zero observed recoveries, and a finite-sample recovery bound in a fresh registered episode. In two audits, on SQLite optimization and virtual catalyst control, the authors report zero recoveries across 96 episodes and an upper bound of 0.0468. Optional feedback tests found truthful feedback recovered targets while neutral feedback did not, under the paper’s registered null calibration. HF Daily Papers' note
score 5