BayesAME: Bayesian Active Model Evaluation
BayesAME chooses how many benchmark items to evaluate by stopping only when its estimate is stable enough.
The paper frames model evaluation as a sequential Bayesian coreset problem, with uncertainty tracked as items are added. It uses historical model performance to model latent item-group abilities, then selects new items by information gain. The process stops when both estimate fluctuation and uncertainty fall below user-set thresholds. The authors report stronger results than sequential versions of existing methods, and argue that non-random coreset selection beats random selection in their experiments. ArXiv · AI/CL/LG's note
The paper frames model evaluation as a sequential Bayesian coreset problem, with uncertainty tracked as items are added. It uses historical model performance to model latent item-group abilities, then selects new items by information gain. The process stops when both estimate fluctuation and uncertainty fall below user-set thresholds. The authors report stronger results than sequential versions of existing methods, and argue that non-random coreset selection beats random selection in their experiments. ArXiv · AI/CL/LG's note
score 4