Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents
A new GB/T-Bench test shows leading LLMs still trail human experts on rule-heavy standards review.
The paper builds a benchmark from 488 China GB/T standard documents, generating 7,306 traceable review errors across 25 error types. Its evaluation requires models to identify the exact location, review dimension, and error type. Across 14 LLMs, the best model scored 0.3280 CMCS, compared with 0.6640 for experts. The authors’ GB/T-Reviewer multi-agent framework lifted the best score to 0.5094 by coordinating specialized review skills. ArXiv · AI/CL/LG's note
The paper builds a benchmark from 488 China GB/T standard documents, generating 7,306 traceable review errors across 25 error types. Its evaluation requires models to identify the exact location, review dimension, and error type. Across 14 LLMs, the best model scored 0.3280 CMCS, compared with 0.6640 for experts. The authors’ GB/T-Reviewer multi-agent framework lifted the best score to 0.5094 by coordinating specialized review skills. ArXiv · AI/CL/LG's note
score 4