Multi-Row, Multi-Span Distant Supervision For Table+Text Question

Vishwajeet Kumar, Saneem Chemmengath, Yash P. Gupta, Jaydeep Sen, Samarth Bharadwaj, Soumen Chakrabarti, Pan, FeiFei

Abstract

Question answering (QA) over tables and linked text, also called TextTableQA, has witnessed significant research in recent years, as tables are often found embedded in documents along with related text. HybridQA and OTT-QA are the two best-known TextTableQA datasets, with questions that are best answered by combining information from both table cells and linked text passages. A common challenge in both datasets, and TextTableQA in general, is that the training instances include just the question and answer, where the gold answer may match not only multiple table cells across table rows but also multiple text spans within the scope of a table row and its associated text. This leads to a noisy multi-instance training regime. We present MITQA, a transformer-based TextTableQA system that is explicitly designed to cope with distant supervision along both these axes, through a multi-instance loss objective, together with careful curriculum design. Our experiments show that the proposed multi-instance distant supervision approach helps MITQA get state-of-the-art results beating the existing baselines for both HybridQA and OTT-QA, putting MITQA at the top of HybridQA leaderboard with best EM and F1 scores on a held out test set.

Introduction: The Challenge of Spurious Matches

Most modern Question Answering (QA) systems excel at either pure text or pure tables. However, real-world data is often hybrid: a Wikipedia page might contain a table of "Tallest Buildings" where answering a query requires synthesizing a table cell with a linked paragraph about that specific building. This domain is known as TextTableQA.

The primary obstacle in training these systems is distant supervision. Training data typically provides only a (Question, Answer) pair without specifying which row or text span contains the actual evidence. In datasets like HybridQA, nearly 40% of questions have the answer string appearing in multiple rows, and 34.5% have it in multiple text spans. These "spurious matches" create a noisy training environment where a model might incorrectly learn to associate an answer with irrelevant context.

Example of the spurious match problem where the answer '2018' appears in multiple rows and passages, creating noise for training.

Methodology: The MITQA Framework

MITQA (Multi-Instance Transformer QA) is a modular pipeline designed to "denoise" this supervision through a multi-instance learning (MIL) approach. The system processes data in several distinct stages:

  1. TableSplitter & PassageFilter: To accommodate the memory constraints of Transformer models, the system treats each table row and its linked text as a single "retrieval unit." A query-sensitive filter ranks and prunes these units to ensure only the most relevant context is processed.
  2. Multi-Instance RowRetriever: Instead of standard cross-entropy loss, which penalizes a model for not selecting every row containing the answer string, MITQA employs an MIL loss. This objective requires the model to assign a high score to at least one correct row in a "bag" of candidates, effectively ignoring spurious matches in other rows.
  3. Multi-Span AnswerExtractor: To pinpoint the exact answer within a row, MITQA uses a two-stage denoising process. It first trains on "easy" cases (single matches) and then uses that model to identify the most likely "true" span in complex, multi-match cases.
  4. Joint Row+Span Reranker (RSR): A final module combines confidence scores from both the retriever and extractor to select the best overall candidate.
The MITQA architecture pipeline, illustrating the flow from TableSplitting to the final Joint Reranker.

Experimental Results

MITQA was evaluated on two major benchmarks, achieving significant gains over previous state-of-the-art (SOTA) baselines:

  • HybridQA: MITQA reached the top of the leaderboard with 65.5 Exact Match (EM) and 72.7 F1 scores, representing a ~21% absolute improvement over the original HYBRIDER baseline.
  • OTT-QA: In the open-domain setting, which requires retrieving tables from a corpus of 400K candidates, MITQA delivered a ~10% absolute improvement over previous benchmarks.

Ablation studies confirmed that the "row-first" retrieval strategy and the Multi-Instance Loss were critical, providing a ~12% F1 boost compared to methods that attempt to find spans across the entire table simultaneously.

Conclusion

MITQA demonstrates that explicitly modeling the noise in distant supervision is key to advancing TextTableQA. By treating potential matches as a "bag" rather than individual gold labels, the model successfully avoids the pitfalls of spurious evidence. Future work will focus on making the pipeline fully end-to-end differentiable and extending the system's capabilities to handle complex numerical reasoning and aggregation queries.

Summary updated

Abstract

Question answering (QA) over tables and linked text, also called TextTableQA, has witnessed significant research in recent years, as tables are often found embedded in documents along with related text. HybridQA and OTT-QA are the two best-known TextTableQA datasets, with questions that are best answered by combining information from both table cells and linked text passages. A common challenge in both datasets, and TextTableQA in general, is that the training instances include just the question and answer, where the gold answer may match not only multiple table cells across table rows but also multiple text spans within the scope of a table row and its associated text. This leads to a noisy multi instance training regime. We present MITQA, a transformer-based TextTableQA system that is explicitly designed to cope with distant supervision along both these axes, through a multi-instance loss objective, together with careful curriculum design. Our experiments show that the proposed multi-instance distant supervision approach helps MITQA get state-of-the-art results beating the existing baselines for both HybridQA and OTT-QA, putting MITQA at the top of HybridQA leaderboard with best EM and F1 scores on a held out test set.
BibTeX
@inproceedings{kumar2023multi,
  author = {Vishwajeet Kumar and Saneem Chemmengath and Yash P. Gupta and Jaydeep Sen and Samarth Bharadwaj and Soumen Chakrabarti and Pan, FeiFei},
  title = {{Multi-Row, Multi-Span Distant Supervision For Table+Text Question}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.449},
}
Figure