Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
PLES is pitched as a cheaper way to infer LLM tuning rules before paying for full-scale runs.
The paper introduces Power-Law Entropy Search, a cost-aware Bayesian optimization method for estimating how optimal hyperparameters shift with model and data scale. Instead of chasing the best single training run, it chooses experiments that most reduce uncertainty in the scaling-law estimate per unit of compute. The authors say it naturally leans on informative small-scale runs and tested it on synthetic benchmarks, LLM-data surrogates, and real pre-training runs. They report accurate scaling-law estimates with less than one-tenth the compute budget of grid search and other baselines. ArXiv · AI/CL/LG's note
The paper introduces Power-Law Entropy Search, a cost-aware Bayesian optimization method for estimating how optimal hyperparameters shift with model and data scale. Instead of chasing the best single training run, it chooses experiments that most reduce uncertainty in the scaling-law estimate per unit of compute. The authors say it naturally leans on informative small-scale runs and tested it on synthetic benchmarks, LLM-data surrogates, and real pre-training runs. They report accurate scaling-law estimates with less than one-tenth the compute budget of grid search and other baselines. ArXiv · AI/CL/LG's note
score 5