Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models

Shujian Zhang, Chengyue Gong, Xingchao Liu

TL;DR

Passage-Mask (PM) is a novel regularization strategy that prevents retriever-reader models from "lazily" overfitting to the highest-ranked search results. By using a learnable, bi-level optimization process to mask out top-tier passages during training, the model is forced to reason across the entire set of retrieved documents, leading to significant performance gains in open-domain QA, dialogue, and fact verification.

Introduction: The "Top-Rank Trap"

In modern NLP, the "Retriever-Reader" architecture is the gold standard for knowledge-intensive tasks. A Retriever (e.g., DPR) pulls relevant documents from a massive corpus like Wikipedia, and a Reader (e.g., Fusion-in-Decoder or FiD) processes these documents to generate an answer.

However, a significant bottleneck exists: the "Top-Rank Trap." Readers tend to overfit the first few passages returned by the retriever because these results often have high lexical overlap with the query. Consequently, the model learns to rely on them almost exclusively. If the correct answer is buried in the "long tail" (e.g., the 50th passage) or if top results are misleading, the model fails. Standard training does not encourage models to look beyond the most obvious evidence, making them brittle.

Methodology: Passage-Mask (PM)

To address this, we introduce Passage-Mask (PM), a mechanism that strategically "hides" certain passages from the reader during training.

Overview of the Passage-Mask (PM) architecture, illustrating how top-ranked passages are masked to force the model to reason across all retrieved documents.

Key Technical Highlights

  • Learnable Masking: Unlike standard Dropout, which is random and uniform, PM learns which passages to mask. It specifically targets top-ranked passages to "desensitize" the model to their influence.
  • Bi-level Optimization: Training is treated as a nested optimization problem. The "inner" loop trains model parameters ($\theta$) to minimize training loss, while the "outer" loop optimizes mask parameters ($w$) to minimize validation loss. This ensures the mask learns to challenge the model effectively.
  • Efficiency: Using a "one-shot" approach with a momentum-assisted gradient estimator, PM updates the mask and model jointly with negligible overhead (~1.6% increase in training time and ~2.7% more GPU memory).
  • Softmax Relaxation: To make the discrete choice of masking differentiable, a Softmax relaxation is applied over a set of mask candidates.

Experimental Results

The Passage-Mask strategy was evaluated across three distinct domains, consistently outperforming baselines:

  • Open-Domain QA: Improved the Exact Match (EM) score of the FiD-base model from 50.1 to 51.3 on Natural Questions and from 69.3 to 69.9 on TriviaQA.
  • Dialogue: On Wizard of Wikipedia (WoW), PM achieved an F1 score of 18.7, significantly outperforming the FiD base (17.1).
  • Fact Verification: Reached a new State-of-the-Art (SOTA) accuracy of 66.5% on FaVIQ-Ambig, up from 65.7%.

Key Insights

  • Desensitization: Masking obvious passages forces the model to find evidence in lower-ranked documents, improving its ability to reason over the entire set of 100+ passages.
  • Robustness: PM-trained models are more resilient to retrieval errors. Even if top-ranked passages are removed or permuted during testing, performance remains high.
  • Superiority over Randomness: Learnable masking identifies specific ranking positions that cause overfitting more effectively than vanilla dropout or random passage removal.

Conclusion

Passage-Mask provides a low-cost, high-impact regularization technique for retriever-reader pipelines. By forcing models to look beyond the most obvious search results, it fosters more robust reasoning and achieves state-of-the-art performance across multiple knowledge-intensive NLP tasks. Future work will explore more flexible search spaces for masks and deeper analysis of internal attention shifts.

Summary updated

Abstract

Retriever-reader models achieve competitive performance across many different NLP tasks such as open question answering and dialogue conversations. In this work, we notice these models easily overfit the top-rank retrieval passages and standard training fails to reason over the entire retrieval passages. We introduce a learnable passage mask mechanism which desensitizes the impact from the top-rank retrieval passages and prevents the model from overfitting. Controlling the gradient variance with fewer mask candidates and selecting the mask candidates with one-shot bi-level optimization, our learnable regularization strategy enforces the answer generation to focus on the entire retrieval passages. Experiments on different tasks across open question answering, dialogue conversation, and fact verification show that our method consistently outperforms its baselines. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.
BibTeX
@inproceedings{zhang2022passage,
  author = {Shujian Zhang and Chengyue Gong and Xingchao Liu},
  title = {{Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=gkI3J2FxxTQ},
}
Figure