Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
Most agents produced correct programs; the gap was how much faster they made them.
Accelerate Bench tests agents on 32 CPU optimization tasks drawn from scientific applications and supercomputing competitions. Among 16 model-and-harness pairings, the top two achieved at least 2× speedups on 22 or more tasks; the other fourteen did so on 8–17. Controlled replays showed identical source code running at 0.29× to 2.12× baseline speed depending only on build and runtime settings. One code change delivered a 1.01× or 6.4× speedup depending on another change.
HF Daily Papers' note
Accelerate Bench tests agents on 32 CPU optimization tasks drawn from scientific applications and supercomputing competitions. Among 16 model-and-harness pairings, the top two achieved at least 2× speedups on 22 or more tasks; the other fourteen did so on 8–17. Controlled replays showed identical source code running at 0.29× to 2.12× baseline speed depending only on build and runtime settings. One code change delivered a 1.01× or 6.4× speedup depending on another change.
HF Daily Papers' note
score 5