TL;DR
MM-ReportBench is a new, rigorous benchmark designed to evaluate how well AI models generate professional multimodal reports—combining fluent text with accurate charts—directly from complex tables. By introducing a "dual-judge" evaluation framework, the researchers expose that even top-tier models like DeepSeek-R1 and Gemini-2.5-Pro struggle with numerical accuracy and genuine analytical discovery, often hallucinating facts in nearly 40% of their reports.
What Problem Does This Paper Address?
In the corporate world, data analysis isn't just about answering a single question; it’s about synthesizing raw tables into structured, multimodal reports that include narrative analysis and supporting visualizations. While Large Language Models (LLMs) have become adept at basic table reasoning, existing benchmarks fail to capture the complexity of professional reporting.
Current evaluations typically suffer from three gaps:
- Single-Modality Focus: They look at text or charts in isolation, ignoring the interplay between them.
- The "Halo Effect": Using a single LLM judge often leads to inflated scores where high-quality prose masks poor-quality visuals (or vice versa).
- Lack of Verifiability: Many benchmarks rely on external retrieval for fact-checking rather than grounding claims directly in the provided tabular data.
Methods
The authors propose MM-ReportBench, which utilizes a sophisticated pipeline for both dataset construction and evaluation:
- Reference Construction via MCTS: To create high-quality "ground truth" reports, the authors used a hybrid pipeline. They employed Monte Carlo Tree Search (MCTS) to explore a vast action space (planning chapters, identifying visualization tasks, generating charts, and refining narratives). This search was guided by a multi-dimensional reward function and followed by rigorous human curation.
- Dual-Judge Evaluation Framework: To prevent score inflation, the benchmark splits evaluation between two specialized agents:
- T-Judge: Evaluates text for structural completeness, grounded insight verification (using SQL to check numbers against source tables), and novelty analysis.
- V-Judge: Assesses visualizations for data fidelity, expressiveness, aesthetics, and technical correctness.
- Cross-Modal Consistency (CMC): A specific module that ensures the text and charts actually support one another.
Datasets
The benchmark consists of 386 tasks based on 185 real-world business tables across six domains: Finance, Healthcare, Manufacturing, Education, Retail, and IT Operations.
- Complexity: Includes tables with over 50,000 cells and complex hierarchical headers.
- Bilingual: Features both Chinese and English data.
- Reference Standards: Each task includes an expert-curated report with standardized structures and verifiable key insight points.
Key Insights and Ideas
- Visual Thinking Improves Factuality: Interestingly, models that generate charts tend to have higher textual factuality scores. The authors suggest that the process of creating a visualization may "force" the model to better verify its data, improving the overall narrative.
- The Single-Judge Trap: The study found that using a single "unified" judge (like GPT-4o) consistently inflates scores by 8–12 points. Decoupling the evaluation of text and visuals is essential to prevent one modality from camouflaging failures in the other.
- Novelty is the Final Frontier: Most models are excellent at paraphrasing existing data but fail miserably at "Novelty Analysis"—the ability to discover non-obvious, actionable insights that aren't just restatements of the table.
- Domain Sensitivity: Performance is not uniform. Models perform best in Retail and IT but struggle significantly in Finance and Healthcare due to complex temporal data and dense accounting metrics.
Key Results
The researchers tested 20 state-of-the-art models, including specialized deep research agents:
- Top Performer: DeepSeek-R1 (code-augmented) achieved the highest overall score of 68.9, followed closely by Gemini-2.5-Pro at 66.8.
- Human Baseline: Humans scored 92.3, highlighting a significant remaining gap for AI.
- Hallucination Rates: Numerical hallucinations were found in 38.5% of all generated reports.
- Visualization Bottleneck: Even top models struggled with "Technical Correctness" in charts (e.g., missing axis labels or unreadable legends), with V-Judge scores hovering around 80 while human-level visuals are near perfect.
Limitations and Future Work
The authors acknowledge several areas for improvement:
- Context Windows: Extremely large tables still cause "truncation effects" where models lose context and produce incomplete analyses.
- Insight Depth: Current models rely heavily on "trivial insights" (e.g., "sales increased") without explaining why or providing deeper context.
- Future Directions: The authors aim to improve numerical faithfulness and develop evaluation protocols that are even more robust against "reward hacking" by policy-trained models.
For a detailed look at how the construction pipeline works, see Figure 1: Overview of the construction pipeline
Summary updated