Abstract
Dense retrieval models are often hindered by the presence of "false negatives"—relevant passages incorrectly labeled as irrelevant during training. This issue is exacerbated by hard negative sampling, where models are inadvertently penalized for correctly identifying relevant but unlabeled content. This paper introduces two complementary solutions: the Contrastive Confidence Regularizer (CCR), a theoretically grounded loss function adaptation, and the Passage Sieve, a model-agnostic data-cleaning algorithm. Experimental results on Natural Questions (NQ), Trivia QA (TQ), and MSMARCO demonstrate that these methods achieve state-of-the-art performance while improving training efficiency and robustness to noise.
1. Introduction
Dense retrieval has become a cornerstone of modern open-domain Question Answering (QA) and information retrieval systems. By encoding queries and passages into a shared vector space, these models can identify relevant information based on semantic similarity. Training these encoders typically relies on contrastive learning, which requires a balance of positive (relevant) and negative (irrelevant) samples.
However, large-scale datasets often suffer from incomplete annotations. When human annotators miss relevant passages, these "false negatives" (FNs) are treated as irrelevant. The problem intensifies with "hard negative sampling," a technique designed to improve model discriminative power by selecting negatives that are highly similar to the query. If a hard negative is actually a missed positive, the model is forced to "push away" relevant information, leading to degraded performance and training instability.
2. Methodology
The authors propose a dual approach to address the false negative problem: a loss-based regularization and a data-centric filtering mechanism.
2.1 Contrastive Confidence Regularization (CCR)
The CCR framework adapts the "peer loss" concept from label-noise robust machine learning to the Noise Contrastive Estimation (NCE) loss used in retrieval. Standard NCE loss can lead to overfitting on noisy labels. CCR mitigates this by subtracting an expectation term—the average loss over a distribution of passages—from the standard loss.
Mathematically, this regularizer penalizes the model when it exhibits excessive confidence in its predictions on potentially noisy data. Theoretical analysis suggests that minimizing this regularized loss is equivalent to minimizing the loss on a clean dataset under specific noise conditions. This provides a robust mathematical foundation for handling label noise without requiring explicit labels for every passage.
2.2 The Passage Sieve Algorithm
To extend these benefits to models that do not utilize NCE loss, the authors developed the Passage Sieve. This is a model-agnostic pre-processing pipeline:
- Robust Initialization: A standard dual-encoder (e.g., DPR) is trained using the CCR loss to create a noise-robust "teacher" model.
- Scoring and Filtering: This robust model scores all potential hard negatives in the training set.
- Thresholding: If a negative passage's score exceeds a threshold (significantly higher than a random-guess baseline), it is flagged as a likely false negative.
- Refined Training: These flagged passages are discarded, and the resulting "sieved" dataset is used to train the final downstream model (e.g., AR2).
3. Experimental Evaluation
The proposed methods were evaluated on three prominent benchmarks:
- Natural Questions (NQ): Google search queries paired with Wikipedia passages.
- Trivia QA (TQ): A large-scale dataset of trivia questions.
- MSMARCO Passage Ranking: A massive dataset based on Bing queries, known for its high rate of unlabeled positives.
4. Results and Discussion
The application of the Passage Sieve to the state-of-the-art AR2 model yielded significant improvements across all benchmarks. On the NQ dataset, Recall@5 improved from 77.9 to 79.2. On MSMARCO, the MRR@10 increased from 39.5 to 39.8.
Beyond raw performance, the research highlights several key advantages:
- Efficiency: The Passage Sieve adds only approximately 10% to total training time, making it far more efficient than cross-encoder-based denoising methods.
- Data Efficiency: Models trained with the sieve achieved better performance using only 5 hard negatives than standard models achieved with 15. This reduction in required samples significantly lowers GPU memory and computational overhead.
- Batch Size Robustness: While traditional dense retrieval models often require very large batch sizes to remain stable, CCR-enhanced models maintained high performance even with smaller batches (e.g., 64).
5. Conclusion and Future Work
This work demonstrates that addressing the false negative problem through confidence regularization and data sieving can substantially improve dense retrieval. By grounding the approach in peer loss theory, the authors provide a principled way to handle noise that outperforms heuristic methods. Future research may explore the use of second-order statistics for even greater robustness and the application of these techniques to a broader range of architectures, such as late-interaction models.
Summary updated