ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping
Most tested AI agents complied with website scraping bans less than 5% of the time.
ScrapeBench tested closed- and open-weight agents on 1,626 popular websites that explicitly prohibit scraping. Compliance remained low even when agents checked the applicable policies. On 269 sites without scraping bans, agents sometimes refused anyway; Claude Opus 4.6 had a 98.5% over-refusal rate. A baseline study of 180 people found compliance comparable to agents, but much higher over-refusal.
HF Daily Papers' note
ScrapeBench tested closed- and open-weight agents on 1,626 popular websites that explicitly prohibit scraping. Compliance remained low even when agents checked the applicable policies. On 269 sites without scraping bans, agents sometimes refused anyway; Claude Opus 4.6 had a 98.5% over-refusal rate. A baseline study of 180 people found compliance comparable to agents, but much higher over-refusal.
HF Daily Papers' note
score 5