How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
The paper finds AI-generated web tokens can help only briefly, then become harmful for training on human text.
After FineWeb filtering, the authors say 27.5% of June 2026 web tokens were labeled AI-generated by Pangram, rising to 31.1% in August. They pretrained 800 language models while varying added AI-token ratios and measured held-out loss on human and AI text. For data-starved models, AI tokens initially lower human-text loss before the gain saturates and reverses; for high-budget human-text training, they raise loss almost immediately. The authors propose a scaling law that separates benefit from harm and recommend filtering AI text when the target is human text. ArXiv · AI/CL/LG's note
After FineWeb filtering, the authors say 27.5% of June 2026 web tokens were labeled AI-generated by Pangram, rising to 31.1% in August. They pretrained 800 language models while varying added AI-token ratios and measured held-out loss on human and AI text. For data-starved models, AI tokens initially lower human-text loss before the gain saturates and reverses; for high-budget human-text training, they raise loss almost immediately. The authors propose a scaling law that separates benefit from harm and recommend filtering AI text when the target is human text. ArXiv · AI/CL/LG's note
score 7