TL;DR
MITQA is a novel framework for question answering over hybrid table-and-text data that tackles the challenge of "distant supervision"—where training data only provides question-answer pairs without specifying the exact cell or text span. By introducing multi-instance loss functions and a multi-span training curriculum, MITQA successfully filters out training noise to achieve state-of-the-art performance on major benchmarks.
What Problem Does This Paper Address?
Real-world documents frequently mix structured tables with unstructured text. Answering questions over this hybrid data (known as TextTableQA) requires models to seamlessly combine information from both sources.
However, training these models is notoriously difficult because datasets typically only provide the question and the final answer string, omitting the exact location of the answer. This creates a tricky distant supervision problem: the answer string might appear multiple times across different rows, or multiple times within text passages linked to a single row.
As shown in Figure 1, naively training a model on all these matches introduces massive noise, as many text matches are completely spurious and irrelevant to the actual question.
Methods
The authors present MITQA, a system that breaks down the hybrid QA task into a sequential pipeline designed to handle this two-dimensional noise (multi-row and multi-span):
- TableSplitter & PassageFilter: Tables are segmented into rows. Because linked text passages can easily exceed a Transformer’s token limit (typically 512 tokens), a query-sensitive
PassageFilteruses Sentence-BERT to rank and prune passages, ensuring the most relevant text is retained. - RowRetriever: A BERT-based sequence classifier identifies the correct row. To handle multiple matching rows, it uses a multi-instance loss function. Instead of forcing the model to score all matching rows highly, the loss function only penalizes the model if it fails to score at least one of the potential gold rows highly. This is paired with a curriculum learning strategy that starts training on "easy" single-match cases before moving to multi-match cases.
- AnswerExtractor: This module extracts the final answer span. It uses a Multi-Span Training (MST) paradigm: an initial model is trained on clean, single-match instances and is then used to "denoise" multi-match instances by selecting the most probable gold span.
- Joint Row+Span Reranker (RSR): A final module that performs a grid search to optimally combine the confidence scores of both the RowRetriever and AnswerExtractor, selecting the best overall answer.
For open-domain settings, MITQA incorporates a TableRetriever (dense passage retrieval) and a RowPassageLinker (which uses GPT-2 to generate context and expand queries).
Datasets
The model was evaluated on two primary TextTableQA datasets:
- HybridQA: A closed-domain dataset where the relevant table and its linked Wikipedia passages are provided alongside the question.
- OTT-QA: An open-domain dataset containing over 400K tables and 5 million passages, requiring the system to retrieve the correct tables and text from scratch before answering.
Key Insights and Ideas
- The Scale of the Noise: Distant supervision noise is a massive hurdle in hybrid datasets. In HybridQA, roughly 40% of training instances suffer from multiple matching rows, and 34.5% contain multiple matching spans.
- Row-First beats Cell-First: Previous architectures (like HYBRIDER) focused on individual cells as the fundamental unit of retrieval. MITQA demonstrates that treating the entire row plus its linked passages as the retrieval unit drastically simplifies the reasoning chain.
- Denoising via Self-Training: Using an initial model trained on "easy" data to label "hard" data acts as an effective data-programming curriculum, preventing the model from learning from spurious text matches.
Key Results
MITQA achieved state-of-the-art results on both benchmarks, significantly outperforming previous baselines:
- HybridQA: On the test set, MITQA achieved a 64.3% Exact Match (EM) and 71.9% F1 score, representing a massive ~21% absolute improvement over the original HYBRIDER baseline.
- OTT-QA: On the open-domain test set, MITQA achieved 36.4% EM and 41.9% F1, beating the best previous baseline by roughly 10% absolute.
Ablation studies confirmed that every component—especially the Multi-Instance Loss (MIL) and the Joint Row+Span Reranker (RSR)—contributed incrementally to these gains.
Limitations and Future Work
- Error Propagation: Because MITQA relies on a modular pipeline rather than a fully end-to-end trainable architecture, errors made early in the pipeline (e.g., selecting the wrong row) inevitably sabotage the final answer extraction.
- Domain Specificity: The benchmarks used are heavily based on Wikipedia. Further experiments are required to see if MITQA adapts well to highly specialized domains like Finance, Healthcare, or Retail.
- Lack of Aggregation: MITQA is designed for span extraction and cell lookup; it cannot currently handle complex numerical or aggregation queries (e.g., "What is the total sum of...").
Summary updated