Abstract
Machine Reading at Scale (MRS) is a challenging task in which a system is given an input query and is asked to produce a precise output by "reading" information from a large knowledge base. The task has gained popularity with its natural combination of information retrieval (IR) and machine comprehension (MC). Advancements in representation learning have led to separated progress in both IR and MC; however, very few studies have examined the relationship and combined design of retrieval and comprehension at different levels of granularity, for development of MRS systems. In this work, we give general guidelines on system design for MRS by proposing a simple yet effective pipeline system with special consideration on hierarchical semantic retrieval at both paragraph and sentence level, and their potential effects on the downstream task. The system is evaluated on both fact verification and open-domain multihop QA, achieving state-of-the-art results on the leaderboard test sets of both FEVER and HOTPOTQA. To further demonstrate the importance of semantic retrieval, we present ablation and analysis studies to quantify the contribution of neural retrieval modules at both paragraph-level and sentence-level, and illustrate that intermediate semantic retrieval modules are vital for not only effectively filtering upstream information and thus saving downstream computation, but also for shaping upstream data distribution and providing better data for downstream modeling. Code/data made publicly available at: https://github.com/easonnie/semanticRetrievalMRS
Introduction
Machine Reading at Scale (MRS) requires systems to efficiently extract precise answers from massive corpora like Wikipedia. Traditionally, this involves a two-step process: Information Retrieval (IR) to find relevant documents and Machine Comprehension (MC) to extract the answer. However, a significant challenge remains in the "misalignment" between these stages. Standard IR often prioritizes recall, leaving the MC "reader" overwhelmed by irrelevant noise. This paper argues that hierarchical semantic retrieval—filtering information at both the paragraph and sentence levels—is essential for bridging this gap and achieving high performance.
System Architecture
The proposed system utilizes a unified, four-stage hierarchical pipeline designed to progressively refine the information provided to the downstream reader:
- Term-Based Retrieval: Uses standard TF-IDF and keyword matching to identify candidate paragraphs from the entire Wikipedia corpus.
- Paragraph-Level Neural Retrieval: A BERT-based model scores candidate paragraphs, narrowing the selection to a high-probability subset.
- Sentence-Level Neural Retrieval: A second BERT-based model selects specific supporting sentences from the filtered paragraphs.
- Downstream Modeling: A task-specific BERT model (QA or NLI) processes only the concatenated supporting sentences to generate the final output.
Crucially, each module is trained on the data distribution generated by its predecessor, mitigating the "distributional shift" that often degrades multi-stage pipeline performance.
Experimental Setup
The system was evaluated on two prominent benchmarks:
- HotpotQA (Fullwiki): A multi-hop QA dataset requiring reasoning across multiple Wikipedia articles.
- FEVER: A fact verification dataset where claims must be labeled as Supported, Refuted, or Not Enough Info based on Wikipedia evidence.
Key Results and Analysis
The hierarchical approach yielded substantial improvements across both tasks. On HotpotQA, the system achieved 45.32% Answer EM and 25.14% Joint EM, doubling the Joint EM of the previous state-of-the-art. On FEVER, the system reached a score of 67.26, representing a 3% absolute improvement over previous systems.
Ablation studies underscored the necessity of paragraph filtering. Removing the paragraph-level neural retrieval caused HotpotQA Joint EM to plummet from 26.6% to a mere 0.34%, demonstrating that sentence-level retrieval alone cannot effectively process the noise from initial term-based searches.
Discussion: Retrieval as Data Shaping
A core insight of this work is that retrieval serves as a "data shaper." By providing a cleaner, more relevant training and inference distribution, semantic retrieval significantly enhances the downstream reader's effectiveness. The authors also noted varying sensitivity to noise: while QA models are relatively robust, FEVER models are highly sensitive, where irrelevant sentences often lead to incorrect "Not Enough Info" classifications. This highlights the importance of precise evidence extraction for fact verification tasks.
Conclusion
This research demonstrates that hierarchical semantic retrieval is a critical component of high-performing MRS systems. By treating retrieval as a core semantic task rather than a simple pre-processing step, the proposed pipeline achieves state-of-the-art results. Future work will explore joint optimization of the pipeline and improved modeling of uncertainty to further mitigate error cascading.
Summary updated