Reference-based Weak Supervision for Answer Sentence Selection using Web Data

Vivek Krishnamurthy, Thuy Vu, Alessandro MoschittiEMNLP 2021 · DOI

Introduction

Answer Sentence Selection (AS2) is a fundamental task in retrieval-based Question Answering (QA). Given a question and a set of candidate sentences, the goal is to identify the sentence that correctly answers the question. While Transformer-based models like BERT and RoBERTa have significantly advanced the field, their performance is heavily dependent on large-scale labeled datasets. Manually labeling millions of sentence pairs is prohibitively expensive and slow. To overcome this bottleneck, researchers from Amazon Alexa AI and UCLA proposed Reference-based Weak Supervision (RWS), an automated pipeline designed to harvest high-quality training data from the open web.

The Reference-based Weak Supervision (RWS) Pipeline

The core innovation of this work is the RWS pipeline, which leverages existing question-answer pairs (references) to automatically label new candidates found on the web. This process transforms the vast, unorganized data of the web into a structured training resource for AS2 models.

The Reference-based Weak Supervision pipeline: from web retrieval to automatic labeling and thresholding.

The pipeline consists of four primary stages:

  1. Retrieval: Starting with a known question-reference pair, the system queries an index of over 100 million web documents (derived from Common Crawl) to find relevant content.
  2. Candidate Selection: Sentences are extracted from the retrieved documents. An initial "off-the-shelf" reranker identifies the top 25 most promising candidates.
  3. Automatic Labeling (AVA): Instead of human annotators, the system uses a semantic evaluator called AVA. AVA compares the web-crawled candidate against the original "reference" answer. If the candidate is semantically equivalent to the reference within the context of the question, it is labeled as "correct."
  4. Thresholding: To ensure data quality, only pairs that receive high confidence scores from the evaluator are retained for the final dataset.

Integration with TANDA

The generated RWS data is integrated into the TANDA (Transfer and Adapt) framework. TANDA involves a two-step fine-tuning process: first, the model is trained on a large, general dataset (Transfer), and then it is fine-tuned on a smaller, domain-specific dataset (Adapt).

Integration of RWS data into the TANDA (Transfer and Adapt) framework for fine-tuning BERT-based models.

By introducing RWS data into the "Transfer" phase, the authors provide the model with a broader "supervision capacity." This allows the model to learn complex linguistic patterns and factual relationships from diverse web sources before specializing in a target benchmark like WikiQA or TREC-QA.

Experimental Results

The effectiveness of the RWS approach was evaluated using standard AS2 benchmarks. The researchers also introduced AS2-NQ, a larger version of the ASNQ dataset, and WQA, a large-scale internal dataset.

Key findings include:

  • New State-of-the-Art: Models enhanced with RWS data set a new record on the WikiQA benchmark, achieving 90.1% P@1 (Precision at 1) and 92.9% MAP (Mean Average Precision).
  • Significant Error Reduction: On the TREC-QA dataset, the RWS pipeline helped a RoBERTa-Large model improve absolute accuracy from 94.1% to 95.6%, which corresponds to a 25% relative error reduction.
  • Overcoming Data Saturation: The study demonstrated that simply adding more human-labeled data from the same distribution (e.g., moving from ASNQ to AS2-NQ) yields diminishing returns. In contrast, the diverse "weak" supervision provided by the web-crawled RWS data successfully pushed model performance beyond previous plateaus.

Conclusion

Reference-based Weak Supervision offers a scalable solution to the data scarcity problem in AS2. By automating the labeling process using semantic evaluators and web-scale data, RWS enables the training of more robust and accurate QA models without the need for exhaustive human intervention. Future work may extend this reference-based paradigm to other information retrieval and natural language processing tasks, further reducing the reliance on manual annotation.

Summary updated

BibTeX
@inproceedings{krishnamurthy2021reference,
  author = {Vivek Krishnamurthy and Thuy Vu and Alessandro Moschitti},
  title = {{Reference-based Weak Supervision for Answer Sentence Selection using Web Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.363},
}
Figure