Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization
The verification method accepted the fewest incorrect statements among the methods tested on three labeled datasets.
It combines LLM-based semantic review with mechanical numerical and vacuity checks, detecting errors that LLM judges missed. The authors used it with their Extract-Verify-Proof pipeline to build a partial Lean formalization of the NIST Digital Library of Mathematical Functions. Expert review also identified translation errors and source defects in existing datasets. The abstract gives no numerical error rates.
ArXiv · AI/CL/LG's note
It combines LLM-based semantic review with mechanical numerical and vacuity checks, detecting errors that LLM judges missed. The authors used it with their Extract-Verify-Proof pipeline to build a partial Lean formalization of the NIST Digital Library of Mathematical Functions. Expert review also identified translation errors and source defects in existing datasets. The abstract gives no numerical error rates.
ArXiv · AI/CL/LG's note
score 5