Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models
Some frontier models appear to reproduce exact published molecular benchmark values, not just estimate them.
The paper audits 22 models across 12 molecular regression benchmarks and reports verbatim retrieval on more than half the models for five datasets. Higher reasoning settings increased retrieval flags by 89% on the same molecules and prompts. Attempts to disrupt retrieval still left some strong models recognizing transformed SMILES paired with original labels. ArXiv · AI/CL/LG's note
The paper audits 22 models across 12 molecular regression benchmarks and reports verbatim retrieval on more than half the models for five datasets. Higher reasoning settings increased retrieval flags by 89% on the same molecules and prompts. Attempts to disrupt retrieval still left some strong models recognizing transformed SMILES paired with original labels. ArXiv · AI/CL/LG's note
score 5