Introduction
Answer Sentence Selection (AS2) is a fundamental task in retrieval-based Question Answering (QA). Given a question and a set of candidate sentences, the goal is to identify the sentence that correctly answers the question. While Transformer-based models like BERT and RoBERTa have significantly advanced the field, their performance is heavily dependent on large-scale labeled datasets. Manually labeling millions of sentence pairs is prohibitively expensive and slow. To overcome this bottleneck, researchers from Amazon Alexa AI and UCLA proposed Reference-based Weak Supervision (RWS), an automated pipeline designed to harvest high-quality training data from the open web.
The Reference-based Weak Supervision (RWS) Pipeline
The core innovation of this work is the RWS pipeline, which leverages existing question-answer pairs (references) to automatically label new candidates found on the web. This process transforms the vast, unorganized data of the web into a structured training resource for AS2 models.
The pipeline consists of four primary stages:
- Retrieval: Starting with a known question-reference pair, the system queries an index of over 100 million web documents (derived from Common Crawl) to find relevant content.
- Candidate Selection: Sentences are extracted from the retrieved documents. An initial "off-the-shelf" reranker identifies the top 25 most promising candidates.
- Automatic Labeling (AVA): Instead of human annotators, the system uses a semantic evaluator called AVA. AVA compares the web-crawled candidate against the original "reference" answer. If the candidate is semantically equivalent to the reference within the context of the question, it is labeled as "correct."
- Thresholding: To ensure data quality, only pairs that receive high confidence scores from the evaluator are retained for the final dataset.
Integration with TANDA
The generated RWS data is integrated into the TANDA (Transfer and Adapt) framework. TANDA involves a two-step fine-tuning process: first, the model is trained on a large, general dataset (Transfer), and then it is fine-tuned on a smaller, domain-specific dataset (Adapt).
By introducing RWS data into the "Transfer" phase, the authors provide the model with a broader "supervision capacity." This allows the model to learn complex linguistic patterns and factual relationships from diverse web sources before specializing in a target benchmark like WikiQA or TREC-QA.
Experimental Results
The effectiveness of the RWS approach was evaluated using standard AS2 benchmarks. The researchers also introduced AS2-NQ, a larger version of the ASNQ dataset, and WQA, a large-scale internal dataset.
Key findings include:
- New State-of-the-Art: Models enhanced with RWS data set a new record on the WikiQA benchmark, achieving 90.1% P@1 (Precision at 1) and 92.9% MAP (Mean Average Precision).
- Significant Error Reduction: On the TREC-QA dataset, the RWS pipeline helped a RoBERTa-Large model improve absolute accuracy from 94.1% to 95.6%, which corresponds to a 25% relative error reduction.
- Overcoming Data Saturation: The study demonstrated that simply adding more human-labeled data from the same distribution (e.g., moving from ASNQ to AS2-NQ) yields diminishing returns. In contrast, the diverse "weak" supervision provided by the web-crawled RWS data successfully pushed model performance beyond previous plateaus.
Conclusion
Reference-based Weak Supervision offers a scalable solution to the data scarcity problem in AS2. By automating the labeling process using semantic evaluators and web-scale data, RWS enables the training of more robust and accurate QA models without the need for exhaustive human intervention. Future work may extend this reference-based paradigm to other information retrieval and natural language processing tasks, further reducing the reliance on manual annotation.
Summary updated