When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems
Offline benchmarks and live A/B tests can rank models differently because users change their prompts in response to each model.
The paper distinguishes performance on a shared prompt distribution from performance on the prompts each model induces through interaction. It shows that standard offline and live-test estimators target those different quantities. It characterizes when rankings reverse and gives a sufficient condition for agreement. A stylized linear model also shows that user adaptation can increase estimation variance, supporting the need to measure both model performance and changes in user behavior.
ArXiv · AI/CL/LG's note
The paper distinguishes performance on a shared prompt distribution from performance on the prompts each model induces through interaction. It shows that standard offline and live-test estimators target those different quantities. It characterizes when rankings reverse and gives a sufficient condition for agreement. A stylized linear model also shows that user adaptation can increase estimation variance, supporting the need to measure both model performance and changes in user behavior.
ArXiv · AI/CL/LG's note
score 5