Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models

Shujian Zhang, Chengyue Gong, Xingchao Liu

Abstract

Retriever-reader models achieve competitive performance across many different NLP tasks such as open question answering and dialogue conversations. In this work, we notice these models easily overfit the top-rank retrieval passages and standard training fails to reason over the entire retrieval passages. We introduce a learnable passage mask mechanism which desensitizes the impact from the top-rank retrieval passages and prevents the model from overfitting. Controlling the gradient variance with fewer mask candidates and selecting the mask candidates with one-shot bi-level optimization, our learnable regularization strategy enforces the answer generation to focus on the entire retrieval passages. Experiments on different tasks across open question answering, dialogue conversation, and fact verification show that our method consistently outperforms its baselines. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.

Introduction

In modern Natural Language Processing (NLP), the "retriever-reader" architecture—exemplified by models like Fusion-in-Decoder (FiD)—has become the standard for knowledge-intensive tasks. These systems operate by retrieving a set of relevant documents from a massive corpus (such as Wikipedia) and then employing a reader model to synthesize these documents into a coherent answer.

However, a significant bottleneck exists: rank-related overfitting. Because gold-standard answers are frequently located in the top-ranked retrieved passages, reader models often learn to "lazy-read," focusing almost exclusively on the first few results. This dependency becomes a liability when the retriever fails to place the correct information at the very top. If the necessary evidence is buried at rank #10 or lower, standard models often fail to identify it. This paper introduces Passage-Mask (PM), a learnable regularization strategy designed to force models to reason over the entire set of retrieved context.

The Passage-Mask (PM) Mechanism

Passage-Mask addresses the "lazy reader" problem by strategically hiding top-ranked passages during the training phase. Unlike standard regularization techniques like Dropout or random masking, which are agnostic to document importance, PM is specifically designed to counter ranking bias.

Learnable Masking and Bi-level Optimization

Instead of masking passages at random, PM utilizes a learnable hyper-parameter to determine which passages to hide. The training is framed as a bi-level optimization problem where the reader model parameters and the mask parameters are optimized jointly. To ensure this process remains computationally feasible, the authors employ a one-shot search strategy with a momentum-assisted gradient estimator. This approach allows the model to learn an optimal masking strategy with minimal overhead—adding only approximately 1.6% to training time and 2.7% to GPU memory usage.

Candidate Selection

To maintain training stability and reduce gradient variance, the authors define a discrete set of "mask candidates" (for example, masking the 1st and 3rd passages) rather than allowing the system to mask arbitrary combinations. This constrained search space ensures that the regularization remains effective without introducing excessive noise into the training process.

Experimental Results

The researchers evaluated Passage-Mask across four diverse benchmarks: Natural Questions (NQ) and TriviaQA for Open-Domain QA, FaVIQ-Ambig (A) for Fact Verification, and Wizard of Wikipedia (WoW) for Knowledge-Grounded Dialogue.

Performance Gains

Passage-Mask consistently outperformed standard training baselines across all tasks:

  • Open-Domain QA: On Natural Questions, PM improved the Exact Match (EM) score of FiD-base from 50.1 to 51.3 and FiD-large from 54.4 to 55.3.
  • Fact Verification: On FaVIQ-A, the model achieved a new State-of-the-Art (SOTA) accuracy of 66.5, surpassing the previous best of 65.7.
  • Dialogue: On Wizard of Wikipedia, the F1 score increased from 17.1 to 18.7, indicating significantly better grounded conversation capabilities.

Confirming the "Lazy Reader" Hypothesis

The authors validated the overfitting issue by manually masking the top passage of a traditionally pre-trained FiD model during testing. Performance plummeted (e.g., from 50.1 to 44.5 on NQ), proving that the model was not effectively utilizing the other 99 passages provided. In contrast, models trained with PM showed much higher robustness when top passages were removed, confirming they had learned to aggregate information from the entire retrieved set.

Conclusion and Future Work

Passage-Mask serves as a powerful, "plug-and-play" regularization module that can be integrated into existing retriever-reader checkpoints during fine-tuning. By forcing the model to look beyond the top-ranked results, PM mitigates ranking bias and improves reasoning across the full context.

While effective, the authors note that PM primarily addresses ranking bias and may not fully resolve deeper lexical or social biases inherent in training data. Future research may explore expanding the mask candidate search space or utilizing Passage-Mask to provide direct feedback to the retriever, potentially improving the initial document ranking itself.

Summary updated

Abstract

Retriever-reader models achieve competitive performance across many different NLP tasks such as open question answering and dialogue conversations. In this work, we notice these models easily overfit the top-rank retrieval passages and standard training fails to reason over the entire retrieval passages. We introduce a learnable passage mask mechanism which desensitizes the impact from the top-rank retrieval passages and prevents the model from overfitting. Controlling the gradient variance with fewer mask candidates and selecting the mask candidates with one-shot bi-level optimization, our learnable regularization strategy enforces the answer generation to focus on the entire retrieval passages. Experiments on different tasks across open question answering, dialogue conversation, and fact verification show that our method consistently outperforms its baselines. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.
BibTeX
@inproceedings{zhang2022passage,
  author = {Shujian Zhang and Chengyue Gong and Xingchao Liu},
  title = {{Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=gkI3J2FxxTQ},
}
Figure