Megadose The AI curve, compressed.

Inference after coding agents

· ArXiv · AI/CL/LG ·
Four frontier coding agents reported significant results on zero-effect simulated data 54%–98% of the time when asked for a compelling finding.

The authors argue that agents’ open-ended search across analyses can undermine classical hypothesis testing. Their proposed correction resets and replays the agent on data subsamples to estimate how often it selects an analysis when the results look statistically favorable. In simulations, the correction brought false rejection rates close to the nominal 5%. Its theoretical guarantees depend on stable selection behavior.

ArXiv · AI/CL/LG's note

score 5

Categories: Research