When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis
The paper argues that long agent runs hit a measurable point where more tokens stop paying off as well as fresh independent attempts.
The authors introduce “Elo-per-token” analysis to compare progress across open-ended tasks with different scoring scales. In tests on four agents and four benchmarks, agents initially gained Elo faster than independent sampling, then saw marginal returns fade below that reference. They define a scaling inflection point and show that, on one benchmark, splitting 100M tokens into parallel sessions beat one long run by 264 Elo. HF Daily Papers' note
The authors introduce “Elo-per-token” analysis to compare progress across open-ended tasks with different scoring scales. In tests on four agents and four benchmarks, agents initially gained Elo faster than independent sampling, then saw marginal returns fade below that reference. They define a scaling inflection point and show that, on one benchmark, splitting 100M tokens into parallel sessions beat one long run by 264 Elo. HF Daily Papers' note
score 5