Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking
Rubric4Setwise turns evaluation criteria into selection signals, outperforming tested rerankers with fewer documents and search rounds.
The paper argues that retrieval for LLMs should judge document sets, not just individually relevant documents. Its SetwiseEvalKit benchmark uses about 28,000 rubrics across three levels and nine dimensions to test short- and long-form scenarios. Across 12 rerankers, the best covered no more than 45%, with cross-document coordination especially weak. HF Daily Papers' note
The paper argues that retrieval for LLMs should judge document sets, not just individually relevant documents. Its SetwiseEvalKit benchmark uses about 28,000 rubrics across three levels and nine dimensions to test short- and long-form scenarios. Across 12 rerankers, the best covered no more than 45%, with cross-document coordination especially weak. HF Daily Papers' note
score 5