S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answering

Fangyu Lei, Xiang Li, Yifan Wei, Shizhu He, Yiming Huang, Jun Zhao, Kang Liu

Abstract

Answering multi-hop questions over hybrid factual knowledge from the given text and table (TextTableQA) is a challenging task. Existing models mainly adopt a retriever-reader framework, which have several deficiencies, such as noisy labeling in training retriever, insufficient utilization of heterogeneous information over text and table, and deficient ability for different reasoning operations. In this paper, we propose a three-stage TextTableQA framework S3HQA, which comprises of retriever, selector, and reasoner. We use a retriever with refinement training to solve the noisy labeling problem. Then, a hybrid selector considers the linked relationships between heterogeneous data to select the most relevant factual knowledge. For the final stage, instead of adapting a reading comprehension module like in previous methods, we employ a generation-based reasoner to obtain answers. This includes two approaches: a row-wise generator and an LLM prompting generator. The experimental results demonstrate that our method achieves competitive results in the few-shot setting. When trained on the full dataset, our approach outperforms all baseline methods, ranking first on the HybridQA leaderboard.

Introduction

In the real world, information is rarely stored in just one format. To answer a question like "Who is the higher scoring athlete from the cities of Eugene and Walnut?", a system must look at a table to find athletes from those cities, follow hyperlinks to Wikipedia passages to verify city locations, and then perform a numerical comparison. This is known as TextTableQA.

Existing models typically use a "retriever-reader" architecture, but they suffer from three major bottlenecks:

  1. Noisy Labeling: During training, if an answer appears in multiple places in the text/table, the retriever is often "weakly supervised" with noisy labels, leading to poor retrieval accuracy.
  2. Information Silos: Models often fail to exploit the rich links (hyperlinks) between table cells and their corresponding descriptive passages.
  3. Reasoning Rigidity: Most "readers" are extraction-based—they try to find a specific "span" of text. This fails for questions requiring calculations, counting, or complex comparisons.

Methodology: The S3HQA Framework

The authors propose a three-stage pipeline called $S^3HQA$ designed to bridge the gap between structured tables and unstructured text.

The S3HQA framework overview, illustrating the three-stage pipeline: Retriever with refinement training, Hybrid Selector, and Generation-based Reasoner.

Stage 1: Retriever with Refinement Training

To solve the noise problem, the training data is split. The retriever is first trained on "clean" instances (where the answer appears only once). This "refined" model then generates pseudo-labels for the noisier parts of the dataset, significantly improving the quality of the evidence retrieved.

Stage 2: Hybrid Selector

This stage acts as the "glue" between heterogeneous data types. It uses the question type to decide how to navigate the table-text graph. For "bridge" questions, it follows hyperlinks from cells to passages. For "comparison" or "count" questions, it aggregates multiple rows to provide a broader context to the reasoner.

Stage 3: Generation-based Reasoner

Instead of just highlighting a word in a sentence, $S^3HQA$ uses a Seq2Seq model (BART-large) to generate the answer. This allows the model to handle diverse operations like counting or comparing values. Additionally, the authors explored LLM Prompting, being the first to apply Large Language Models (like GPT-3.5) with Chain-of-Thought (CoT) prompting to the TextTableQA task.

Experimental Results

The model was evaluated on HybridQA, a large-scale benchmark containing over 60,000 instances. The dataset is particularly challenging because it requires multi-hop reasoning where the evidence is split between Wikipedia tables and their linked passages.

Performance comparison of S3HQA against baseline models on the HybridQA dataset. S3HQA achieves state-of-the-art results across Table, Passage, and Total metrics.

As shown in the results, $S^3HQA$ achieved 68.4% Exact Match (EM) and 75.3% F1 score on the HybridQA test set, outperforming previous top models like MAFiD and MITQA. Ablation studies proved that removing "Refinement Training" caused a 3.2% drop in retrieval accuracy, confirming that handling label noise is critical. Furthermore, in a 2-shot setting, GPT-3.5 with Chain-of-Thought achieved a 72.1% F1 score, coming remarkably close to fully supervised models without extensive task-specific training.

Conclusion

$S^3HQA$ demonstrates that a three-stage approach combining refined retrieval, hybrid selection, and generative reasoning can effectively master complex TextTableQA tasks. While the model currently ranks first on the HybridQA leaderboard, future work will focus on improving the interpretability of the reasoning path and enhancing generalizability across different types of hybrid datasets.

Summary updated

Abstract

Answering multi-hop questions over hybrid factual knowledge from the given text and table (TextTableQA) is a challenging task. Existing models mainly adopt a retriever-reader framework, which have several deficiencies, such as noisy labeling in training retriever, insufficient utilization of heterogeneous information over text and table, and deficient ability for different reasoning operations. In this paper, we propose a three-stage TextTableQA framework S3HQA, which comprises of retriever, selector, and reasoner. We use a retriever with refinement training to solve the noisy labeling problem. Then, a hybrid selector considers the linked relationships between heterogeneous data to select the most relevant factual knowledge. For the final stage, instead of adapting a reading comprehension module like in previous methods, we employ a generation-based reasoner to obtain answers. This includes two approaches: a row-wise generator and an LLM prompting generator~(first time used in this task). The experimental results demonstrate that our method achieves competitive results in the few-shot setting. When trained on the full dataset, our approach outperforms all baseline methods, ranking first on the HybridQA leaderboard.
BibTeX
@inproceedings{lei2023s,
  author = {Fangyu Lei and Xiang Li and Yifan Wei and Shizhu He and Yiming Huang and Jun Zhao and Kang Liu},
  title = {{S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.147},
}
Figure