Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs
The evaluation pipeline cut false-positive verdicts on natural-language proofs from 23% to 6%, the authors report.
The pipeline assesses natural-language proofs, computer algebra code and Lean formalizations for semantic and logical correctness. Compared with one-shot LLM judgment, it also raised accuracy on supporting code from 46% to 58%. Ramanujan Challenge submissions included correct proofs conceptually distinct from the official solutions and a proof of one of the challenge’s published open conjectures. The benchmark covers research-level number theory, including unpublished results and open conjectures.
ArXiv · AI/CL/LG's note
The pipeline assesses natural-language proofs, computer algebra code and Lean formalizations for semantic and logical correctness. Compared with one-shot LLM judgment, it also raised accuracy on supporting code from 46% to 58%. Ramanujan Challenge submissions included correct proofs conceptually distinct from the official solutions and a proof of one of the challenge’s published open conjectures. The benchmark covers research-level number theory, including unpublished results and open conjectures.
ArXiv · AI/CL/LG's note
score 5