Multi-Row, Multi-Span Distant Supervision For Table+Text Question

Vishwajeet Kumar, Yash Gupta, Saneem Chemmengath, Jaydeep Sen, Soumen Chakrabarti, Samarth Bharadwaj, FeiFei Pan

TL;DR

MITQA is a novel framework for question answering over hybrid table-and-text data that tackles the challenge of "distant supervision"—where training data only provides question-answer pairs without specifying the exact cell or text span. By introducing multi-instance loss functions and a multi-span training curriculum, MITQA successfully filters out training noise to achieve state-of-the-art performance on major benchmarks.

What Problem Does This Paper Address?

Real-world documents frequently mix structured tables with unstructured text. Answering questions over this hybrid data (known as TextTableQA) requires models to seamlessly combine information from both sources.

However, training these models is notoriously difficult because datasets typically only provide the question and the final answer string, omitting the exact location of the answer. This creates a tricky distant supervision problem: the answer string might appear multiple times across different rows, or multiple times within text passages linked to a single row.

Figure 1: An instance of question answering over hybrid context of table and text (from HybridQA). Gold answer in correct context is highlighted in blue and gold answer appearing in irrelevant context is highlighted in red. The context used to arrive at the answer in the correct passage is shaded in yellow. The relevant row to be retrieved is shaded green and irrelevant rows are shaded red.

As shown in Figure 1, naively training a model on all these matches introduces massive noise, as many text matches are completely spurious and irrelevant to the actual question.

Methods

The authors present MITQA, a system that breaks down the hybrid QA task into a sequential pipeline designed to handle this two-dimensional noise (multi-row and multi-span):

  1. TableSplitter & PassageFilter: Tables are segmented into rows. Because linked text passages can easily exceed a Transformer’s token limit (typically 512 tokens), a query-sensitive PassageFilter uses Sentence-BERT to rank and prune passages, ensuring the most relevant text is retained.
  2. RowRetriever: A BERT-based sequence classifier identifies the correct row. To handle multiple matching rows, it uses a multi-instance loss function. Instead of forcing the model to score all matching rows highly, the loss function only penalizes the model if it fails to score at least one of the potential gold rows highly. This is paired with a curriculum learning strategy that starts training on "easy" single-match cases before moving to multi-match cases.
  3. AnswerExtractor: This module extracts the final answer span. It uses a Multi-Span Training (MST) paradigm: an initial model is trained on clean, single-match instances and is then used to "denoise" multi-match instances by selecting the most probable gold span.
  4. Joint Row+Span Reranker (RSR): A final module that performs a grid search to optimally combine the confidence scores of both the RowRetriever and AnswerExtractor, selecting the best overall answer.

For open-domain settings, MITQA incorporates a TableRetriever (dense passage retrieval) and a RowPassageLinker (which uses GPT-2 to generate context and expand queries).

Figure 2: MITQA system sketch. TableRetriever and RowPassageLinker are not shown.

Datasets

The model was evaluated on two primary TextTableQA datasets:

  • HybridQA: A closed-domain dataset where the relevant table and its linked Wikipedia passages are provided alongside the question.
  • OTT-QA: An open-domain dataset containing over 400K tables and 5 million passages, requiring the system to retrieve the correct tables and text from scratch before answering.

Key Insights and Ideas

  • The Scale of the Noise: Distant supervision noise is a massive hurdle in hybrid datasets. In HybridQA, roughly 40% of training instances suffer from multiple matching rows, and 34.5% contain multiple matching spans.
  • Row-First beats Cell-First: Previous architectures (like HYBRIDER) focused on individual cells as the fundamental unit of retrieval. MITQA demonstrates that treating the entire row plus its linked passages as the retrieval unit drastically simplifies the reasoning chain.
  • Denoising via Self-Training: Using an initial model trained on "easy" data to label "hard" data acts as an effective data-programming curriculum, preventing the model from learning from spurious text matches.

Key Results

MITQA achieved state-of-the-art results on both benchmarks, significantly outperforming previous baselines:

  • HybridQA: On the test set, MITQA achieved a 64.3% Exact Match (EM) and 71.9% F1 score, representing a massive ~21% absolute improvement over the original HYBRIDER baseline.
  • OTT-QA: On the open-domain test set, MITQA achieved 36.4% EM and 41.9% F1, beating the best previous baseline by roughly 10% absolute.

Ablation studies confirmed that every component—especially the Multi-Instance Loss (MIL) and the Joint Row+Span Reranker (RSR)—contributed incrementally to these gains.

Limitations and Future Work

  • Error Propagation: Because MITQA relies on a modular pipeline rather than a fully end-to-end trainable architecture, errors made early in the pipeline (e.g., selecting the wrong row) inevitably sabotage the final answer extraction.
  • Domain Specificity: The benchmarks used are heavily based on Wikipedia. Further experiments are required to see if MITQA adapts well to highly specialized domains like Finance, Healthcare, or Retail.
  • Lack of Aggregation: MITQA is designed for span extraction and cell lookup; it cannot currently handle complex numerical or aggregation queries (e.g., "What is the total sum of...").

Summary updated

Abstract

Question answering (QA) over tables and linked text, also called TextTableQA, has witnessed significant research in recent years, as tables are often found embedded in documents along with related text. HybridQA and OTT-QA are the two best-known TextTableQA datasets, with questions that are best answered by combining information from both table cells and linked text passages. A common challenge in both datasets, and TextTableQA in general, is that the training instances include just the question and answer, where the gold answer may match not only multiple table cells across table rows but also multiple text spans within the scope of a table row and its associated text. This leads to a noisy multi instance training regime. We present MITQA, a transformer-based TextTableQA system that is explicitly designed to cope with distant supervision along both these axes, through a multi-instance loss objective, together with careful curriculum design. Our experiments show that the proposed multi-instance distant supervision approach helps MITQA get state-of-the-art results beating the existing baselines for both HybridQA and OTT-QA, putting MITQA at the top of HybridQA leaderboard with best EM and F1 scores on a held out test set.
BibTeX
@inproceedings{kumar2023multi,
  author = {Vishwajeet Kumar and Yash Gupta and Saneem Chemmengath and Jaydeep Sen and Soumen Chakrabarti and Samarth Bharadwaj and FeiFei Pan},
  title = {{Multi-Row, Multi-Span Distant Supervision For Table+Text Question}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.449},
}
Figure