Post-hoc Alignment of LLM-judges to Human Judgment Distribution
The paper says LLM judges can match human majorities while still missing how humans disagree.
Across five datasets, the authors found near human-level hard-label performance on most tasks, but weak prediction of soft-label human judgment distributions. They propose NAPHA, a lightweight post-hoc alignment method that routes examples by entropy class to specialized alignment models. It consistently improved soft-label prediction across models and datasets, especially on high-entropy cases. ArXiv · AI/CL/LG's note
Across five datasets, the authors found near human-level hard-label performance on most tasks, but weak prediction of soft-label human judgment distributions. They propose NAPHA, a lightweight post-hoc alignment method that routes examples by entropy class to specialized alignment models. It consistently improved soft-label prediction across models and datasets, especially on high-entropy cases. ArXiv · AI/CL/LG's note
score 4