Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers
Reordering the same candidates can change what an LLM scorer keeps, even when ranking quality looks unchanged.
The paper tests LLM scorers that assign individual scores while seeing multiple candidates in one prompt. In passage reranking, five trained scorers within 0.010 nDCG@10 kept reordered result sets with only 0.66-0.84 overlap. Prompt-time fixes did not remove the dependence. The authors propose order-consistency SFT, which penalizes score disagreement across orderings while preserving ranking quality. HF Daily Papers' note
The paper tests LLM scorers that assign individual scores while seeing multiple candidates in one prompt. In passage reranking, five trained scorers within 0.010 nDCG@10 kept reordered result sets with only 0.66-0.84 overlap. Prompt-time fixes did not remove the dependence. The authors propose order-consistency SFT, which penalizes score disagreement across orderings while preserving ranking quality. HF Daily Papers' note
score 5