MM-ReportBench: A Comprehensive Benchmark for Multimodal Report Generation from Tables

Teng Lin, Nan TangarXiv 2026 · arXiv

TL;DR

MM-ReportBench is a new, rigorous benchmark designed to evaluate how well AI models generate professional multimodal reports—combining fluent text with accurate charts—directly from complex tables. By introducing a "dual-judge" evaluation framework, the researchers expose that even top-tier models like DeepSeek-R1 and Gemini-2.5-Pro struggle with numerical accuracy and genuine analytical discovery, often hallucinating facts in nearly 40% of their reports.

What Problem Does This Paper Address?

In the corporate world, data analysis isn't just about answering a single question; it’s about synthesizing raw tables into structured, multimodal reports that include narrative analysis and supporting visualizations. While Large Language Models (LLMs) have become adept at basic table reasoning, existing benchmarks fail to capture the complexity of professional reporting.

Current evaluations typically suffer from three gaps:

  1. Single-Modality Focus: They look at text or charts in isolation, ignoring the interplay between them.
  2. The "Halo Effect": Using a single LLM judge often leads to inflated scores where high-quality prose masks poor-quality visuals (or vice versa).
  3. Lack of Verifiability: Many benchmarks rely on external retrieval for fact-checking rather than grounding claims directly in the provided tabular data.

Methods

The authors propose MM-ReportBench, which utilizes a sophisticated pipeline for both dataset construction and evaluation:

  • Reference Construction via MCTS: To create high-quality "ground truth" reports, the authors used a hybrid pipeline. They employed Monte Carlo Tree Search (MCTS) to explore a vast action space (planning chapters, identifying visualization tasks, generating charts, and refining narratives). This search was guided by a multi-dimensional reward function and followed by rigorous human curation.
  • Dual-Judge Evaluation Framework: To prevent score inflation, the benchmark splits evaluation between two specialized agents:
  • T-Judge: Evaluates text for structural completeness, grounded insight verification (using SQL to check numbers against source tables), and novelty analysis.
  • V-Judge: Assesses visualizations for data fidelity, expressiveness, aesthetics, and technical correctness.
  • Cross-Modal Consistency (CMC): A specific module that ensures the text and charts actually support one another.
Fig. 2. The Dual-Judge Evaluation Framework of MM-ReportBench. CMC verifies whether textual references to charts and chart-supported narratives are mutually consistent.

Datasets

The benchmark consists of 386 tasks based on 185 real-world business tables across six domains: Finance, Healthcare, Manufacturing, Education, Retail, and IT Operations.

  • Complexity: Includes tables with over 50,000 cells and complex hierarchical headers.
  • Bilingual: Features both Chinese and English data.
  • Reference Standards: Each task includes an expert-curated report with standardized structures and verifiable key insight points.

Key Insights and Ideas

  • Visual Thinking Improves Factuality: Interestingly, models that generate charts tend to have higher textual factuality scores. The authors suggest that the process of creating a visualization may "force" the model to better verify its data, improving the overall narrative.
  • The Single-Judge Trap: The study found that using a single "unified" judge (like GPT-4o) consistently inflates scores by 8–12 points. Decoupling the evaluation of text and visuals is essential to prevent one modality from camouflaging failures in the other.
  • Novelty is the Final Frontier: Most models are excellent at paraphrasing existing data but fail miserably at "Novelty Analysis"—the ability to discover non-obvious, actionable insights that aren't just restatements of the table.
  • Domain Sensitivity: Performance is not uniform. Models perform best in Retail and IT but struggle significantly in Finance and Healthcare due to complex temporal data and dense accounting metrics.
Fig. 3. (1) shows the overall scores of three representative models under Single Judge vs. Dual-Judg. Dual-Judge’s scores are consistently lower, revealing a score inflation of roughly 8–12 points in the single-judge setting. The chart also illustrates the severe performance drop when visual generation is disabled (w/o visuals). (2) is a radar chart of domain-level MM-ReportBench scores for the top three models. DeepSeek-R1 maintains a lead across all domains, but all models exhibit a similar “hard domain” pattern: Finance and Healthcare are consistently more difficult than Retail and IT Operations.

Key Results

The researchers tested 20 state-of-the-art models, including specialized deep research agents:

  • Top Performer: DeepSeek-R1 (code-augmented) achieved the highest overall score of 68.9, followed closely by Gemini-2.5-Pro at 66.8.
  • Human Baseline: Humans scored 92.3, highlighting a significant remaining gap for AI.
  • Hallucination Rates: Numerical hallucinations were found in 38.5% of all generated reports.
  • Visualization Bottleneck: Even top models struggled with "Technical Correctness" in charts (e.g., missing axis labels or unreadable legends), with V-Judge scores hovering around 80 while human-level visuals are near perfect.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Context Windows: Extremely large tables still cause "truncation effects" where models lose context and produce incomplete analyses.
  • Insight Depth: Current models rely heavily on "trivial insights" (e.g., "sales increased") without explaining why or providing deeper context.
  • Future Directions: The authors aim to improve numerical faithfulness and develop evaluation protocols that are even more robust against "reward hacking" by policy-trained models.

For a detailed look at how the construction pipeline works, see Figure 1: Overview of the construction pipeline

Overview of the construction pipeline

Summary updated

Abstract

The ability to generate multimodal reports integrating fluent narratives with faithful visualizations from tabular data, has become a cornerstone capability for modern vision-language models in practical domains such as finance, healthcare, and policy making. However, existing benchmarks are limited by single-modality focus, simplistic metrics, novelty evaluation, and visualization quality. We address these limitations with MM-ReportBench, a multimodal report evaluation benchmark covering 386 tasks across 185 real-world tables. Each task is paired with an expert-curated reference report featuring standardized chapter structures and verifiable key insight points, constructed through a hybrid pipeline of multimodal reports generation based on Monte Carlo Tree Search and rigorous human quality control. For robust evaluation, we design a dual-judge evaluation framework: Text Judge Model assesses textual quality along structural completeness, grounded insight verification with numerical fact-checking against source tables, and novelty analysis that distinguishes genuine analytical discovery from trivial paraphrasing; Visualization Judge Model evaluates generated visualizations across data fidelity, expressiveness, aesthetics, and technical correctness. They also evaluate cross-modal consistency. Extensive experiments on 20 state-of-the-art models reveal critical weaknesses: even the strongest model achieves only 68.9 overall score, with numerical hallucination affecting 38.5% of reports and genuinely novel insights remaining rare. Our ablation studies further demonstrate that single-judge evaluation consistently inflates scores by 8-12 points compared to dual-judge assessment, confirming that rigorous and reliable benchmarking demands multi-faceted, complementary evaluation probes.
BibTeX
@misc{lin2026mm,
  author = {Teng Lin and Nan Tang},
  title = {{MM-ReportBench: A Comprehensive Benchmark for Multimodal Report Generation from Tables}},
  year = {2026},
  eprint = {2611.22514},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22514},
}
Figure