BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction
The benchmark finds frontier systems are still poor at reconstructing the early papers that later became AI research signals.
BackTrend tests whether systems can recover underrecognized problems and emerging methods that preceded 25 mature AI and machine-learning topics. Its reference set has 66 human-validated weak signals, grounded in 2019-2024 publication-frequency trajectories. The authors report that systems often produce plausible but wrong precursors, with the best reaching 10.1% F1 and at most 18.5% Coverage10. More retrieval and web search help only up to a point. ArXiv · AI/CL/LG's note
BackTrend tests whether systems can recover underrecognized problems and emerging methods that preceded 25 mature AI and machine-learning topics. Its reference set has 66 human-validated weak signals, grounded in 2019-2024 publication-frequency trajectories. The authors report that systems often produce plausible but wrong precursors, with the best reaching 10.1% F1 and at most 18.5% Coverage10. More retrieval and web search help only up to a point. ArXiv · AI/CL/LG's note
score 5