Holdout Best-of-N: Unbiased Evaluation and Its Cost
Reusing winner-selection scores can make a Best-of-N system look better than it is.
The paper proves that an exactly unbiased estimator from a fixed score matrix exists only when the selector uses fewer fresh scores than the matrix contains. In the hard case where selection uses `K-1` of `K` scores, Holdout reaches the stated unbiased minimax rate, while biased methods can be faster. For two candidates, the authors derive the minimum-variance unbiased estimator at known variance and show Holdout attains the sharp asymptotic constant without knowing that variance. The fixed-matrix impossibility does not apply if one extra fresh winner score is available. ArXiv · AI/CL/LG's note
The paper proves that an exactly unbiased estimator from a fixed score matrix exists only when the selector uses fewer fresh scores than the matrix contains. In the hard case where selection uses `K-1` of `K` scores, Holdout reaches the stated unbiased minimax rate, while biased methods can be faster. For two candidates, the authors derive the minimum-variance unbiased estimator at known variance and show Holdout attains the sharp asymptotic constant without knowing that variance. The fixed-matrix impossibility does not apply if one extra fresh winner score is available. ArXiv · AI/CL/LG's note
score 5