TRH2TQA: Table Recognition with Hierarchical Relationships to Table Question-Answering on Business Table Images

Pongsakorn Jirachanchaisiri, Nam Tuan Ly, Atsuhiro TakasuWACV 2025

TL;DR

This paper introduces TRH2TQA, a novel pipeline for answering questions from business table images. It tackles the challenge of hierarchical table structures by first recognizing table content and structure, then explicitly identifying hierarchical relationships, and finally using this enriched information for question answering, achieving state-of-the-art performance on the VQAonBD 2023 dataset.

What Problem Does This Paper Address?

Extracting actionable insights from business documents, especially financial reports, is crucial for data-driven decision-making. These documents often contain complex tables with intricate structures, including merged cells and hierarchical relationships (e.g., parent-child rows). While visual question answering (VQA) has advanced, answering questions from such visually rich and structured documents, particularly tables, remains a significant challenge. Existing approaches often struggle with the dense, often numerical, and context-dependent nature of tabular data, especially when hierarchical relationships are present, which are vital for understanding the data's meaning.

Methods

TRH2TQA employs a two-stage pipeline approach:

  1. Table Recognition (TR) Module: This module uses a Local MTL-TabNet model to extract both the structure (HTML tags) and textual content of each table cell from an input image. It also predicts cell bounding boxes.
  2. Post-Structure Extraction Module: This module takes the output from the TR module and performs two key tasks:
  3. Header Detection: Identifies header rows using heuristics based on column spans, missing values, and duplicate cell values.
  4. Hierarchical Row Detection: This is the core novelty. It identifies parent-child relationships between rows using algorithms that leverage predicted HTML structure and/or bounding box information (Colspan, Bbox, ColspanBbox). This step enriches the table data by associating parent row information with child rows.
  5. Table Question-Answering (TQA) Module: The structured and hierarchically enriched table data is converted into a dataframe. This dataframe, along with the natural language question, is fed into an encoder-decoder model (TAPEX or OmniTab are used for fine-tuning) to generate the final answer.

Datasets

  • FinTabNet: Used for training the Table Recognition (TR) module. It's a large dataset of complex tables from financial documents, with HTML and cell bounding box annotations.
  • VQAonBD 2023: Used for training and evaluating the Table Question-Answering (TQA) module. This dataset is extracted from FinTabNet and categorizes questions by difficulty.

Key Insights and Ideas

  • Hierarchical relationships are crucial for TQA: The paper strongly demonstrates that explicitly identifying and incorporating hierarchical relationships in tables significantly improves question-answering performance, especially for complex queries.
  • Pipeline approach offers interpretability: While multi-step, the pipeline approach (TR -> Post-Structure Extraction -> TQA) provides better interpretability compared to end-to-end models.
  • Trade-offs in automatic hierarchy detection: While automatic hierarchical row detection is promising, the current implementation shows a slight drop in performance compared to non-automatic methods, suggesting a need for further refinement.
  • TAPEX excels at SQL-like reasoning: TAPEX, with its SQL query execution pre-training, shows a slight edge over OmniTab in handling complex numerical calculations and hierarchical queries, likely due to its specialized training.

Key Results

  • TRH2TQA achieves state-of-the-art performance on the VQAonBD 2023 dataset.
  • The paper reports significant improvements in TQA accuracy when hierarchical relationships are utilized, with TRH2TQA showing an 11.21% increase over OmniTab and a 4.89% increase over TAPEX (non-fine-tuned).
  • For TR on FinTabNet, TEDS scores (measuring structure and content accuracy) are high, with the original Local MTL-TabNet achieving an average TEDS of 0.9866 (structure only) and 0.9518 (structure and content).
  • The study shows a strong linear correlation between TR accuracy (TEDS) and TQA accuracy, highlighting the foundational importance of accurate table recognition.

Limitations and Future Work

  • Scalability: The TR module might face increased computation for larger tables, and the TQA module could experience slower inference with longer sequences.
  • Automatic Hierarchy Detection Refinement: The current automatic hierarchical detection shows a performance drop, indicating a need for further optimization to match or exceed non-automatic methods.
  • Error Propagation: The paper acknowledges the potential for errors from TR to propagate to TQA and suggests implementing feedback loops or making TQA more resilient to minor TR errors.
  • Multi-Hierarchy Detection: Future work aims to introduce logic for detecting multi-level hierarchies to handle even more complex tables.
  • Beyond Textual Hierarchies: Exploring hierarchical information beyond text inputs (e.g., data structures, transformer layers) is also a future direction.
  • Real-world Impact Evaluation: Complementing ANLS with evaluations considering correctness, relevance, and business impact is suggested for future work.

Summary updated

BibTeX
@inproceedings{jirachanchaisiri2025trh,
  author = {Pongsakorn Jirachanchaisiri and Nam Tuan Ly and Atsuhiro Takasu},
  title = {{TRH2TQA: Table Recognition with Hierarchical Relationships to Table Question-Answering on Business Table Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure