Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering

Hao Cheng, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

Abstract

Researchers from Microsoft and Google have demonstrated that how we mathematically frame "distant supervision"—the process of training models on noisy, automatically labeled data—drastically impacts performance in document-level question answering. By systematically optimizing the probability space and combining multiple supervision assumptions into a multi-objective model, they achieved new state-of-the-art results on major benchmarks like TriviaQA and NarrativeQA.

What Problem Does This Paper Address?

In extractive Question Answering (QA), a model must find the specific span of text in a document that answers a question. Ideally, we’d train these models on "gold" data where humans have highlighted the exact answer. However, human labeling is expensive.

To scale up, researchers use Distant Supervision (DS). Instead of precise highlights, we take a question-answer pair (e.g., “Who won the Oscar?” / “Parasite”) and a relevant document. We then assume that any occurrence of the string "Parasite" in that document is a potential answer.

Examples of distant supervision noise in TriviaQA and NarrativeQA datasets.

The challenge? This is incredibly noisy. A document might mention "Parasite" five times, but only one instance actually answers the question. Some mentions might be irrelevant or even misleading (semantic drift). This paper investigates how different probabilistic assumptions about this noise affect a model's ability to learn.

Methods

The authors break down the document-level QA task into three critical axes:

  1. Probability Space: Should the model treat each paragraph independently (Paragraph-level), or should all paragraphs in a document compete for the "best answer" slot in a single joint distribution (Document-level)?
  2. Distant Supervision Assumptions: They test three "hypotheses" (H1, H2, H3) about the labels:
  3. H1 (Aggressive): All mentions of the answer string are correct. (High coverage, high noise).
  4. H2 (Moderate): Every paragraph that contains the answer string has at least one correct mention.
  5. H3 (Selective): Somewhere in the whole document, there is at least one correct mention, but many paragraphs might be total noise. (Low noise, but harder to learn).
  6. Optimization & Inference: They compare Maximum Marginal Likelihood (MML)—which sums the probability of all possible correct spans—against Hard EM, which picks the single most likely span during training.
The model architecture, illustrating the hierarchy from BERT contextualized representations to final string probabilities.

The core technical contribution is a multi-objective model that combines these assumptions, allowing the model to benefit from the high coverage of paragraph-level labels while using document-level logic to "clean" the noise.

Datasets

The models were evaluated on two primary document-level QA benchmarks:

  • TriviaQA (Wiki): Known for having high noise, as answer strings often have many "aliases" or mentions that don't actually answer the specific question.
  • NarrativeQA (Summaries): Based on movie scripts and books. The noise level is lower here because the answer strings are human-generated summaries of the content.

Key Insights and Ideas

  • The "Cleaning" Effect: The H3 (Document-level) assumption acts as a filter. It allows the model to assign zero probability to a paragraph if it decides the mentions within it are just noise. This was crucial for TriviaQA.
  • Inference Matters: For TriviaQA, "Sum" inference (aggregating scores of the same answer string across the document) performed better because correct answers often appear multiple times. For NarrativeQA, "Max" inference was better, as answers are more unique.
  • Position vs. Span: The authors found that position-based MML (calculating the probability of the start and end of a span independently) consistently outperformed span-based approaches. This is likely because the independence assumption simplifies the learning landscape for the BERT encoder.
  • Multi-Objective Synergy: You don't have to choose just one assumption. By training on a combination of H2-P (paragraph-level coverage) and H3-D (document-level cleaning), the model achieves a "best of both worlds" effect.

Key Results

The proposed approach set a new state-of-the-art (SOTA) at the time of publication:

  • TriviaQA-Wiki: Achieved 76.3 F1, outperforming the previous best model by 4.3 points.
  • NarrativeQA: Achieved 62.9 Rouge-L, a 1.7 point improvement over previous SOTA.
  • Verified Subset: On the human-verified portion of TriviaQA, the model reached 85.5 F1, showing that the probabilistic improvements in handling noise translate directly to better "real-world" accuracy.

Limitations and Future Work

The authors acknowledge that their current model still focuses on extractive QA—finding a string that exists verbatim in the text. It does not handle multi-hop reasoning, where a model must combine information from two different parts of a document to form an answer.

Future directions include extending these probabilistic frameworks to open-domain QA (searching across millions of documents) and applying similar "noise-aware" logic to other distantly supervised tasks like relation extraction.

Summary updated

BibTeX
@inproceedings{cheng2020probabilistic,
  author = {Hao Cheng and Ming-Wei Chang and Kenton Lee and Kristina Toutanova},
  title = {{Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=BfQNVy4MTau},
}
Figure