Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization
Agents kept improving in the second half of 135 out of 150 runs, and model rankings changed with evaluation length.
Infrathon tested six models on AI infrastructure optimization tasks with 24-hour budgets, consuming more than 10,000 GPU-hours. The results suggest shorter evaluations can miss gains and changes in relative model performance. Claude Opus 5.5 raised the mean KV-cache compression ratio from the human-designed Quant-VideoGen codec’s 6.82% to 19.86%. Task violations, unreliable testing and premature completion limited sustained improvement.
HF Daily Papers' note
Infrathon tested six models on AI infrastructure optimization tasks with 24-hour budgets, consuming more than 10,000 GPU-hours. The results suggest shorter evaluations can miss gains and changes in relative model performance. Claude Opus 5.5 raised the mean KV-cache compression ratio from the human-designed Quant-VideoGen codec’s 6.82% to 19.86%. Task violations, unreliable testing and premature completion limited sustained improvement.
HF Daily Papers' note
score 6