Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Eight candidates can cost very different amounts depending on how they are batched.
The paper finds that raising candidate count improved GSM8K accuracy for Phi-3-mini and Qwen2.5-1.5B, but that count alone hid the systems cost. With `N = 8`, eight serial generation calls used 4.64-4.86x more gross GPU-device energy than one batched call on A100 GPUs. The serial setup also had 5.77-6.12x the P95 latency. The authors argue evaluations should report generation schedule and GPU-level metrics alongside candidate count and accuracy. HF Daily Papers' note
The paper finds that raising candidate count improved GSM8K accuracy for Phi-3-mini and Qwen2.5-1.5B, but that count alone hid the systems cost. With `N = 8`, eight serial generation calls used 4.64-4.86x more gross GPU-device energy than one batched call on A100 GPUs. The serial setup also had 5.77-6.12x the P95 latency. The authors argue evaluations should report generation schedule and GPU-level metrics alongside candidate count and accuracy. HF Daily Papers' note
score 4