OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened

<p>This story is wild. The short version: OpenAI were running a cybersecurity test against an unreleased model, with the model's guardrail features turned off. Rather than solve the test, the model broke its way out of OpenAI's sandbox, then found exploits to break <em>in</em> to Hugging Face, all so it could cheat on the test by stealing the answers.</p> <p>Along the way it helped make the strongest case yet for how the imbalance of model availability is hurting our ability to secure our software.</p> <h4 id="here-s-what-happened">Here's what happened</h4> <p>We currently have three documents to help us understand what happened here.</p> <ol> <li> <a href="https://arxiv.org/abs/2605.11086">ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?</a> is a paper published on 11th May 2026 describing ExploitGym, a new eval suite for LLM-powered agent systems.</li> <li> <a href="https://huggingface.co/blog/security-incident-july-2026">Security incident disclosure — July 2026</a> by Hugging Face on 16th July 2026 describes how they detected an attack from an "agentic security-research harness - used LLM still not known" that breached some of their systems.</li> <li> <a href="https://openai.com/index/hugging-face-model-evaluation-security-incident/">OpenAI and Hugging Face partner to address security incident during model evaluation</a> from OpenAI on 21st July 2026 confesses that it was <em>their</em> agent harness that did this, and that they're working with Hugging Face to clean up the mess.</li> </ol> <h4 id="exploitgym">ExploitGym</h4> <p>I hadn't seen the <a href="https://arxiv.org/abs/2605.11086">ExploitGym paper</a> before and it's a really interesting one. Authors from UC Berkeley, the Max Planck Institute, UC Santa Barbara, and Arizona State designed a new benchmark for evaluating models on their ability to turn a reported vulnerability into a concrete exploit. OpenAI, Anthropic, and Google provided feedback and helped run the benchmark agains