Abstract
We address the problem of extractive question answering using document-level distant supervision, pairing questions and relevant documents with answer strings. We compare previously used probability space and distant supervision assumptions (assumptions on the correspondence between the weak answer string labels and possible answer mention spans). We show that these assumptions interact, and that different configurations provide complementary benefits. We demonstrate that a multi-objective model can efficiently combine the advantages of multiple assumptions and outperform the best individual formulation. Our approach outperforms previous state-of-the-art models by 4.3 points in F1 on TriviaQA-Wiki and 1.7 points in Rouge-L on NarrativeQA summaries.
Introduction
Training high-quality extractive Question Answering (QA) models typically requires "gold-standard" data where humans highlight the exact start and end positions of an answer within a text. However, collecting such data is prohibitively expensive. A popular alternative is distant supervision (DS), which pairs questions with answer strings (e.g., "Joan Rivers") and relevant documents. The challenge lies in the "needle in a haystack" problem: a document may contain multiple mentions of an answer string, but not all mentions necessarily answer the specific question. This paper investigates how different mathematical models of these uncertainties impact the robustness of QA systems.
Methods: A Probabilistic Framework
The proposed model utilizes a standard BERT-based encoder but focuses innovation on the probabilistic "head." The architecture maps contextualized representations to string-level probabilities by calculating begin/end probabilities, span probabilities, and finally aggregating these into string probabilities.
The authors break the problem down into three dimensions:
- Probability Space: Comparing paragraph-level models (independent paragraphs) vs. document-level models (joint probability space across all paragraphs).
- Distant Supervision Hypotheses: Formalizing how answer mentions are treated (H1, H2, and H3).
- Optimization & Inference: Testing Maximum Marginal Likelihood (MML) vs. Hard EM training objectives and Sum vs. Max aggregation during inference.
Distant Supervision Hypotheses
A core contribution of this work is the formalization of three distant supervision hypotheses regarding the answer mentions ($\mathcal{A}$):
- H1 (All-in): Assumes every mention of an answer string is correct. This provides high coverage but introduces significant noise.
- H2 (Paragraph-level): Assumes every paragraph containing an answer string has at least one correct mention.
- H3 (Document-level): Assumes the document as a whole contains at least one correct mention. This allows the model to effectively "denoise" data by ignoring irrelevant paragraphs.
The "secret sauce" of the approach is a Multi-Objective Formulation, which trains the model to satisfy multiple hypotheses simultaneously, combining the high coverage of H2 with the noise-filtering capabilities of H3.
Key Results and Insights
The study evaluates models on two major benchmarks: TriviaQA (Wikipedia) and NarrativeQA (Summaries). Key findings include:
- Document-level calibration is essential: Normalizing probabilities across the entire document allows for better comparison of candidate answers from disparate sections.
- H3 is a powerful denoiser: For noise-heavy datasets like TriviaQA, the H3 assumption is crucial as it permits the model to assign zero probability to paragraphs that contain the answer string but do not answer the question.
- State-of-the-Art Performance: The approach achieved 76.3 F1 on TriviaQA-Wiki (+4.3 points over previous SOTA) and 62.9 Rouge-L on NarrativeQA (+1.7 points).
Conclusion
This research demonstrates that the mathematical interpretation of noisy labels is as critical as the underlying neural architecture in distantly-supervised QA. By systematically combining different probabilistic assumptions, the authors provide a robust framework for document-level reasoning. Future work may extend these insights to open-domain QA and multi-document synthesis.
Summary updated