TL;DR
Passage-Mask (PM) is a novel regularization strategy that prevents retriever-reader models from "lazily" overfitting to the highest-ranked search results. By using a learnable, bi-level optimization process to mask out top-tier passages during training, the model is forced to reason across the entire set of retrieved documents, leading to significant performance gains in open-domain QA, dialogue, and fact verification.
Introduction: The "Top-Rank Trap"
In modern NLP, the "Retriever-Reader" architecture is the gold standard for knowledge-intensive tasks. A Retriever (e.g., DPR) pulls relevant documents from a massive corpus like Wikipedia, and a Reader (e.g., Fusion-in-Decoder or FiD) processes these documents to generate an answer.
However, a significant bottleneck exists: the "Top-Rank Trap." Readers tend to overfit the first few passages returned by the retriever because these results often have high lexical overlap with the query. Consequently, the model learns to rely on them almost exclusively. If the correct answer is buried in the "long tail" (e.g., the 50th passage) or if top results are misleading, the model fails. Standard training does not encourage models to look beyond the most obvious evidence, making them brittle.
Methodology: Passage-Mask (PM)
To address this, we introduce Passage-Mask (PM), a mechanism that strategically "hides" certain passages from the reader during training.
Key Technical Highlights
- Learnable Masking: Unlike standard Dropout, which is random and uniform, PM learns which passages to mask. It specifically targets top-ranked passages to "desensitize" the model to their influence.
- Bi-level Optimization: Training is treated as a nested optimization problem. The "inner" loop trains model parameters ($\theta$) to minimize training loss, while the "outer" loop optimizes mask parameters ($w$) to minimize validation loss. This ensures the mask learns to challenge the model effectively.
- Efficiency: Using a "one-shot" approach with a momentum-assisted gradient estimator, PM updates the mask and model jointly with negligible overhead (~1.6% increase in training time and ~2.7% more GPU memory).
- Softmax Relaxation: To make the discrete choice of masking differentiable, a Softmax relaxation is applied over a set of mask candidates.
Experimental Results
The Passage-Mask strategy was evaluated across three distinct domains, consistently outperforming baselines:
- Open-Domain QA: Improved the Exact Match (EM) score of the FiD-base model from 50.1 to 51.3 on Natural Questions and from 69.3 to 69.9 on TriviaQA.
- Dialogue: On Wizard of Wikipedia (WoW), PM achieved an F1 score of 18.7, significantly outperforming the FiD base (17.1).
- Fact Verification: Reached a new State-of-the-Art (SOTA) accuracy of 66.5% on FaVIQ-Ambig, up from 65.7%.
Key Insights
- Desensitization: Masking obvious passages forces the model to find evidence in lower-ranked documents, improving its ability to reason over the entire set of 100+ passages.
- Robustness: PM-trained models are more resilient to retrieval errors. Even if top-ranked passages are removed or permuted during testing, performance remains high.
- Superiority over Randomness: Learnable masking identifies specific ranking positions that cause overfitting more effectively than vanilla dropout or random passage removal.
Conclusion
Passage-Mask provides a low-cost, high-impact regularization technique for retriever-reader pipelines. By forcing models to look beyond the most obvious search results, it fosters more robust reasoning and achieves state-of-the-art performance across multiple knowledge-intensive NLP tasks. Future work will explore more flexible search spaces for masks and deeper analysis of internal attention shifts.
Summary updated