MM-ReportBench: A Comprehensive Benchmark for Multimodal Report Generation from Tables
Numerical hallucinations affected 38.5% of reports in tests of 20 models.
MM-ReportBench covers 386 tasks drawn from 185 real-world tables, with expert-curated reference reports and verifiable key insights. Separate text and visualization judges check numerical accuracy, analytical novelty, chart quality and consistency between text and visuals. The strongest model scored 68.9 overall, and genuinely novel insights remained rare. Using a single judge inflated scores by 8–12 points compared with the dual-judge assessment.
ArXiv · AI/CL/LG's note
MM-ReportBench covers 386 tasks drawn from 185 real-world tables, with expert-curated reference reports and verifiable key insights. Separate text and visualization judges check numerical accuracy, analytical novelty, chart quality and consistency between text and visuals. The strongest model scored 68.9 overall, and genuinely novel insights remained rare. Using a single judge inflated scores by 8–12 points compared with the dual-judge assessment.
ArXiv · AI/CL/LG's note
score 4