More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
The paper finds direct-decision models compress ordinal scales, even when their accuracy looks strong.
On ANLI, JEV put 38.8% of predictions and 51.3% of errors into Neutral despite 74.95% accuracy and balanced labels. Across 36 ordinal datasets, final decisions used only 67-76% of the effective gold support, far below performance on nominal tasks. The authors say randomizing candidate order reduced but did not remove the effect. BA-LoRA post-training raised utilization from about 47% to 86% on eight supervised scales, suggesting the bias is learned and adjustable. HF Daily Papers' note
On ANLI, JEV put 38.8% of predictions and 51.3% of errors into Neutral despite 74.95% accuracy and balanced labels. Across 36 ordinal datasets, final decisions used only 67-76% of the effective gold support, far below performance on nominal tasks. The authors say randomizing candidate order reduced but did not remove the effect. BA-LoRA post-training raised utilization from about 47% to 86% on eight supervised scales, suggesting the bias is learned and adjustable. HF Daily Papers' note
score 3