AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games
AV-AIVAT pairs AIVAT variance reduction with anytime-valid confidence sequences so agent evaluations can stop early without breaking the confidence guarantee.
The paper reports a median 54x variance reduction from AIVAT across 15 LLM agent setups and 71,439 paired HUNL hands. At 95% confidence with a ±1 Big Blind target precision, raw outcomes needed a median 74x more hands than AIVAT-corrected outcomes under AsympCS. For exact finite-sample certification, the authors use Empirical-Bernstein confidence sequences and give a structural payoff bound for Leduc hold’em. The stated goal is an auditable stopping rule: stop when the evidence is enough, then let a third party recheck the verdict at that stopping time. ArXiv · AI/CL/LG's note
The paper reports a median 54x variance reduction from AIVAT across 15 LLM agent setups and 71,439 paired HUNL hands. At 95% confidence with a ±1 Big Blind target precision, raw outcomes needed a median 74x more hands than AIVAT-corrected outcomes under AsympCS. For exact finite-sample certification, the authors use Empirical-Bernstein confidence sequences and give a structural payoff bound for Leduc hold’em. The stated goal is an auditable stopping rule: stop when the evidence is enough, then let a third party recheck the verdict at that stopping time. ArXiv · AI/CL/LG's note
score 5