Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit
The paper says a significant LLM-judge bias signal can be produced by the rating scale itself.
The authors argue that double-difference audits on bounded rating scales can mix real preference with unequal censoring near the scale limits. In their pre-registered 990-call audit of a frozen pedagogy judge, the primary learner-profile effect was null: +0.085 points, with p = 0.684. One nominally significant interaction, +0.378 with p = 0.002, was largely reproducible without differential preference, using the observed severity shift and the scale floor. ArXiv · AI/CL/LG's note
The authors argue that double-difference audits on bounded rating scales can mix real preference with unequal censoring near the scale limits. In their pre-registered 990-call audit of a frozen pedagogy judge, the primary learner-profile effect was null: +0.085 points, with p = 0.684. One nominally significant interaction, +0.378 with p = 0.002, was largely reproducible without differential preference, using the observed severity shift and the scale floor. ArXiv · AI/CL/LG's note
score 4