Introduction
Financial analysis represents a significant challenge for numerical reasoning in artificial intelligence. While Large Language Models (LLMs) have demonstrated proficiency in general Question Answering (QA), the domain of corporate financial reporting remains a complex frontier. Existing benchmarks, such as FinQA or TAT-QA, often simplify this task by providing single, "flat" tables paired with limited textual context.
In practice, corporate annual reports are intricate documents containing dozens of hierarchical tables with nested row and column headers. Answering a professional-grade question—such as determining a specific segment's sales increase relative to its funding—requires an analyst to navigate multiple tables, cross-reference data points, and perform multi-step arithmetic. This paper introduces MultiHiertt, a large-scale benchmark designed to reflect this real-world complexity, alongside MT2Net, a novel retrieval-reasoning pipeline.
The MultiHiertt Dataset
The MultiHiertt dataset is a substantial contribution to the field, comprising 10,440 QA pairs derived from 2,513 S&P 500 annual reports. Its design focuses on three pillars of complexity:
- Scale and Scope: Each document contains an average of 3.89 tables and exceeds 1,600 words, far surpassing the scope of previous datasets.
- Structural Complexity: Tables are predominantly hierarchical, requiring models to understand nested relationships (e.g., "Operating Expenses" $\rightarrow$ "Research and Development") to correctly interpret cell values.
- Reasoning Depth: Approximately 71% of the questions require two or more reasoning steps, and nearly half (48.74%) necessitate the integration of information from both structured tables and unstructured text.
Methodology: The MT2Net Framework
To address the challenges of MultiHiertt, the researchers developed MT2Net, a two-stage retrieval-reasoning framework. Given that financial documents often exceed the token limits of standard transformer models, MT2Net first employs a Fact Retrieving Module. This module uses a linearization technique where each table cell is converted into a descriptive sentence that includes its full hierarchical path (e.g., "For Innovation Systems of Segment, sales of product in 2018 is 2,894").
The second stage is the Reasoning Module, which utilizes a RoBERTa-large encoder to process retrieved facts. It features two specialized components:
- Program Sub-module: An LSTM-based generator that produces symbolic mathematical programs (e.g.,
add(number1, number2)), ensuring precise arithmetic execution. - Span Sub-module: Designed for extractive questions where the answer is a specific phrase or entity found within the text.
Results and Analysis
Experimental results indicate that MT2Net significantly outperforms previous state-of-the-art models. It achieved an Exact Match (EM) of 36.22% and an F1 score of 38.43%, compared to 31.72% EM for FinQANet and 17.81% EM for TAGOP. This improvement is largely attributed to the model's superior handling of hierarchical structures through its header-path linearization.
However, a substantial performance gap remains between AI and human experts. Human financial analysts achieved an 83.12% EM on the same test set. Error analysis revealed that the primary bottleneck is retrieval: 64% of model errors occurred because the system failed to identify the correct operands from the lengthy reports. Furthermore, many questions require external financial domain knowledge—such as specific accounting formulas—that is not explicitly provided in the source text.
Conclusion
MultiHiertt establishes a new, more rigorous standard for evaluating numerical reasoning in the financial domain. While MT2Net represents a step forward in managing hierarchical data and multi-hop reasoning, the ~47-point gap between machine and human performance highlights the need for more sophisticated, structure-aware retrieval mechanisms and the integration of external financial expertise into AI models. Future work will likely focus on question decomposition and more robust retrieval strategies to navigate the "labyrinth" of corporate financial reporting.
Summary updated