Robustness of IR Models to Collection Growth
Adding non-relevant documents still degraded every retrieval model the authors tested.
The paper defines robustness as a retriever’s effectiveness not falling when unrelated documents are added to a collection. It tests that by merging collections with negligible topic overlap. Models that depend on the broader collection, such as BM25-style IDF or listwise rerankers, and models that do not both showed degradation. In the tested setup, document-agnostic retrievers were more effective than dependent retrievers, while the rerankers were roughly even. ArXiv · AI/CL/LG's note
The paper defines robustness as a retriever’s effectiveness not falling when unrelated documents are added to a collection. It tests that by merging collections with negligible topic overlap. Models that depend on the broader collection, such as BM25-style IDF or listwise rerankers, and models that do not both showed degradation. In the tested setup, document-agnostic retrievers were more effective than dependent retrievers, while the rerankers were roughly even. ArXiv · AI/CL/LG's note
score 4