RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature
RATIO tests retrieval systems on whether they can find papers that help generate scientific ideas in three specific ways.
The benchmark defines relevance as Address, Broaden, or Specify: finding approaches to problems, more general formulations, or concrete instantiations. It is built from millions of full-text CS papers using discourse-marker distant supervision, then checked with LLM and human vetting. The paper reports that operation-specific fine-tuning improves retrievers, but performance still leaves substantial room for work. ArXiv · AI/CL/LG's note
The benchmark defines relevance as Address, Broaden, or Specify: finding approaches to problems, more general formulations, or concrete instantiations. It is built from millions of full-text CS papers using discourse-marker distant supervision, then checked with LLM and human vetting. The paper reports that operation-specific fine-tuning improves retrievers, but performance still leaves substantial room for work. ArXiv · AI/CL/LG's note
score 5