Abstract
Jie Zhang, Changzai Pan, Sishi Xiong, Kaiwen Wei, Yu Zhao, Xiangyu Li, Jiaxin Peng, Xiaoyan Gu, Jian Yang, Wenhan Chang, Zhenhe Wu, Jiang Zhong, Shuangyong Song, Xuelong Li. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
2025.
Introduction
While Large Language Models (LLMs) have demonstrated proficiency in specific table-related tasks such as TableQA or short-form Table2Text generation, they frequently struggle with the more complex "table-to-report" (T2R) task. This task, central to the work of business analysts, requires transforming large-scale, multi-sheet industrial tables with hierarchical structures into comprehensive, article-level reports. Existing benchmarks often rely on small, clean datasets like Wikipedia tables, which do not reflect the complexity of industrial data. Furthermore, evaluating long-form reports is challenging as traditional metrics like BLEU or ROUGE fail to capture semantic accuracy and logical flow.
Methodology
To address these challenges, the authors developed T2R-bench, a bilingual benchmark, and a multi-faceted evaluation framework. The pipeline for creating this benchmark involves data collection from real-world sources followed by a rigorous annotation process.
The evaluation system comprises three primary criteria:
- Numerical Accuracy Criterion (NAC): Uses code-generation LLMs to verify numerical claims in reports by executing Python code against the source data, effectively mitigating hallucinations.
- Information Coverage Criterion (ICC): Employs Normalized Mutual Information to measure the semantic alignment between generated reports and 5–10 reference keypoints.
- General Evaluation Criterion (GEC): An LLM-as-a-judge approach that assesses reasoning depth, style, practicality, and coherence.
Dataset Characteristics
T2R-bench includes 457 industrial tables across 19 domains (e.g., Finance, Healthcare). It features four distinct table types, including "Extremely Large-Size" tables (up to 700,000 rows) and "Complex Structured" tables with merged cells. The dataset provides 910 high-quality questions and 4,320 annotated keypoints in both Chinese and English. Reports in T2R-bench average 950 words, significantly longer than the 10-20 word answers found in datasets like WikiSQL.
Experimental Results and Insights
Evaluation of 25 state-of-the-art LLMs revealed several key findings:
- Performance Cliff: Model performance drops sharply as table size increases. For instance, DeepSeek-R1's NAC score fell from 74.58% on single tables to 28.43% on massive tables.
- Format Sensitivity: Markdown was found to be the most effective input format for table reasoning, outperforming HTML and JSON.
- Reasoning vs. Retrieval: Errors often stemmed from incorrect operations (e.g., growth rate calculations) across multiple tables rather than simple retrieval failures.
- Human-AI Gap: There remains a nearly 30-point gap between the best-performing model (DeepSeek-R1 at 62.71%) and the human baseline (89.32%).
Conclusion
T2R-bench highlights the limitations of current LLMs in handling industrial-scale tabular data. Future work includes addressing context window truncation, which accounts for 25% of errors, and exploring multimodal "Table-LLMs" that can better interpret complex structures.
Summary updated