Your Benchmark Is an Empirical Measure Over Difficulty
Benchmarks can miss real model improvements when they leave parts of the difficulty range untested.
The authors analyzed 25 benchmarks across seven domains, finding that many overweight particular difficulty levels or leave substantial gaps. They distinguish difficulty distributions that differ from the intended evaluation target from gaps that hide localized training progress. The paper introduces two diagnostic metrics with theoretical guarantees, plus BenchFill, an item-rewriting algorithm that reduces local coverage gaps.
HF Daily Papers' note
The authors analyzed 25 benchmarks across seven domains, finding that many overweight particular difficulty levels or leave substantial gaps. They distinguish difficulty distributions that differ from the intended evaluation target from gaps that hide localized training progress. The paper introduces two diagnostic metrics with theoretical guarantees, plus BenchFill, an item-rewriting algorithm that reduces local coverage gaps.
HF Daily Papers' note
score 5