MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data

Yilun Zhao, Yunxiang Li, Chenying Li, Rui Zhang

Introduction

Numerical reasoning—the ability to interpret data and perform mathematical operations to answer queries—is a cornerstone of financial analysis. While NLP models have shown promise on datasets like FinQA and TAT-QA, these benchmarks often simplify the problem by using single, "flat" tables. In reality, financial reports are complex documents containing multiple tables with nested, hierarchical structures and extensive textual context. To address this gap, researchers have introduced MULTIHIERTT, a large-scale benchmark designed to test models against the messy reality of S&P 500 annual reports.

The MULTIHIERTT Benchmark

The MULTIHIERTT dataset consists of 10,440 QA pairs derived from 2,513 financial documents. Unlike its predecessors, which typically feature one table per document, MULTIHIERTT documents average nearly four tables each. A defining characteristic of these tables is their hierarchical nature, where row and column headers are nested to provide context for specific data points.

Example of hierarchical row and column headers in a financial table.

This complexity introduces significant hurdles for standard models. Approximately 70% of the questions in the dataset require multi-step reasoning, and nearly 25% require "cross-table" reasoning—linking data from different tables to reach an answer.

MT2Net: A Two-Stage Reasoning Framework

To tackle these challenges, the authors proposed MT2Net (Multi-Table-to-Net), a pipeline model designed to handle long-form documents and hierarchical data.

The MT2Net architecture, featuring fact retrieval and program-based reasoning.

The framework operates in two primary stages:

  1. Fact Retrieving Module: Because financial reports exceed the input limits of standard transformers, this module filters the document for relevant information. It "verbalizes" hierarchical table cells by prepending their header paths (e.g., "For Innovation Systems, sales of product in 2018 was 2,894") and uses a BERT-based classifier to select the most relevant facts.
  2. Reasoning Module: This module processes the retrieved facts using two specialized sub-modules. The Program Sub-module generates symbolic math programs (e.g., divide(subtract(x, y), y)) to perform multi-step arithmetic, while the Span Sub-module extracts specific phrases or numbers directly from the text when mathematical operations are not required.

Results and the Human-Machine Gap

Experimental results highlight the difficulty of the MULTIHIERTT benchmark. The MT2Net model, using a RoBERTa-large backbone, achieved a state-of-the-art F1 score of 38.43%. While this significantly outperformed existing baselines like TAGOP and FinQANet (which struggled with the hierarchical data), it remains far below the performance of human experts.

Professional financial analysts achieved an F1 score of 87.03% on the same tasks. The performance of AI models drops sharply as complexity increases; for questions requiring more than three mathematical steps, MT2Net's accuracy fell to just 15.14%. This "human-machine gap" underscores the limitations of current models in maintaining a global view of complex documents and applying domain-specific common sense.

Conclusion

MULTIHIERTT exposes the "frontier" challenges of financial NLP: cross-table integration, the need for domain knowledge, and the limitations of flattening hierarchical data. While the "verbalization" technique used in MT2Net provides a path forward, future research will likely need to explore more advanced table-encoding methods and question decomposition strategies to bridge the gap between automated systems and human expertise.

Summary updated

BibTeX
@inproceedings{zhao2022multihiertt,
  author = {Yilun Zhao and Yunxiang Li and Chenying Li and Rui Zhang},
  title = {{MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=h7ilAN8WEeW},
}
Figure