AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models
Unchanged model pairs drew more “both are bad” votes over time, while their relative rankings held steady.
The working paper examines 1,260,967 blind Chatbot Arena votes from May 2023 to August 2024, restricting comparisons to fixed model versions. Across fourteen pairs frequently evaluated in both years, “both are bad” votes rose from 15.5% to 23.3%. The rise persisted within individual voters and after controls for prompts, language and interface changes. The authors interpret this as evidence of rising evaluation standards, with implications for leaderboards and human feedback used to train models.
ArXiv · AI/CL/LG's note
The working paper examines 1,260,967 blind Chatbot Arena votes from May 2023 to August 2024, restricting comparisons to fixed model versions. Across fourteen pairs frequently evaluated in both years, “both are bad” votes rose from 15.5% to 23.3%. The rise persisted within individual voters and after controls for prompts, language and interface changes. The authors interpret this as evidence of rising evaluation standards, with implications for leaderboards and human feedback used to train models.
ArXiv · AI/CL/LG's note
score 5