Abstract: Yixin Nie, Songhe Wang, Mohit Bansal. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Introduction
Machine Reading at Scale (MRS) is the task of answering queries by searching and comprehending information from massive knowledge bases, such as the entirety of Wikipedia. Traditionally, MRS systems are architected as a two-stage process: Information Retrieval (IR) to locate relevant documents, followed by Machine Comprehension (MC) to extract or verify an answer.
However, a significant gap exists in current designs. Most research focuses on maximizing retrieval recall—ensuring the correct answer is present in the retrieved set—while often neglecting retrieval precision. This paper argues that imprecise retrieval introduces excessive "noise" (irrelevant text), which creates a distribution mismatch between training and inference. This mismatch, or internal covariate shift, fundamentally limits the performance of the downstream reader.
Methodology: A Hierarchical Neural Pipeline
To address these challenges, the authors propose a unified, multi-stage hierarchical pipeline. The system moves beyond simple keyword searches by employing semantic filtering at increasingly granular levels:
- Term-Based Retrieval: A combination of TF-IDF and keyword matching narrows the corpus to a candidate set of paragraphs.
- Paragraph-Level Neural Retrieval: A BERT-based binary classifier scores paragraphs for semantic relevance. Only the top $k_p$ paragraphs are retained.
- Sentence-Level Neural Retrieval: A second BERT module decomposes selected paragraphs into sentences, selecting only the most pertinent evidence.
- Downstream Modeling: A final BERT-based "reader" (either a Question Answering or Natural Language Inference model) processes the refined sentences to produce the final output.
Crucially, the downstream modules are trained on data sampled directly from the upstream retrieval modules. This ensures the reader is robust to the specific types of noise and errors generated by the retriever.
Datasets and Evaluation
The system was evaluated on two major benchmarks requiring intermediate evidence selection:
- FEVER: A fact verification dataset where claims must be supported or refuted using Wikipedia.
- HotpotQA: A multi-hop question-answering dataset requiring reasoning across multiple documents.
Key Insights
Retrieval as a "Data Shaper"
The core contribution of this work is the realization that retrieval modules do more than filter; they define the environment for the downstream model. By improving retriever precision, the input distribution for the "reader" becomes cleaner and more consistent, mitigating performance degradation caused by irrelevant context.
The Necessity of Paragraph Filtering
Ablation studies revealed that paragraph-level neural retrieval is imperative. Without it, the sentence-level retriever is overwhelmed by noise. In HotpotQA, removing paragraph-level filtering caused the Joint Exact Match (EM) score to plummet from 26.60% to a negligible 0.34%.
Task-Specific Sensitivity
The authors observed that different tasks have different sensitivities: HotpotQA (QA) was more sensitive to paragraph-level noise, while FEVER (verification) was more sensitive to sentence-level noise. This suggests that the retrieval hierarchy must be tuned specifically to the downstream task's requirements.
Key Results
The hierarchical approach achieved substantial gains over previous state-of-the-art (SOTA) models. On HotpotQA, the system achieved a 45.32% Answer EM, an 8% absolute improvement. More significantly, it doubled the Joint EM (measuring both answer and supporting facts) to 25.14%.
On FEVER, the system achieved a score of 67.26%, a 3% absolute improvement, while establishing a massive 22-point lead in evidence F1 score over contemporary systems.
Conclusion
This research demonstrates that for Machine Reading at Scale, the quality of intermediate retrieval is as vital as the final comprehension step. Precise, hierarchical semantic retrieval doesn't just provide explainability; it is a critical performance driver that aligns data distributions and allows downstream models to focus on relevant information. Future work may explore joint training of these stages to further harmonize the retriever-reader relationship.
Summary updated