TL;DR
When training dense retrieval models via knowledge distillation, the common wisdom is to aggressively mine "hard negatives." This paper challenges that paradigm, demonstrating that focusing exclusively on hard negatives blinds the student model to the teacher's broader preference structure. By introducing a simple, parameter-free Stratified Sampling strategy that uniformly covers the teacher's entire score spectrum, the authors achieve superior generalization across both in-domain and out-of-domain retrieval benchmarks.
What Problem Does This Paper Address?
In modern Information Retrieval (IR), we often use a two-stage approach: a fast but less accurate dense retriever (bi-encoder) narrows down millions of documents to a few hundred, and a powerful but slow reranker (cross-encoder) scores them precisely. To make the dense retriever smarter, researchers use Knowledge Distillation (KD), where the dense retriever (the student) learns to mimic the continuous score outputs of the reranker (the teacher).
However, how do we choose which negative documents to use during this distillation training? The standard practice relies heavily on heuristics, such as picking the top-$K$ hardest negatives found by an initial retriever. The authors argue that this approach provides a highly biased, narrow view of the teacher's knowledge. If the student only sees extremely difficult negatives, it struggles to map out robust decision boundaries for varying levels of difficulty, ultimately hurting its ability to generalize to new, unseen domains.
Methods
To isolate the impact of score distributions from the messy variables of dynamic data mining, the authors created a controlled candidate pool of 200 negative documents per query (100 top-retrieved documents + 100 randomly sampled documents).
Instead of just picking the top-scoring documents from this pool, they proposed Stratified Sampling. The process is straightforward and deterministic:
- Normalize Scores: Apply query-level min-max normalization to the teacher's scores so they span from 0 to 1.
- Set Quantile Anchors: Evenly space out $K$ target probabilities (anchors) across the distribution.
- Select Closest Matches: For each anchor, select the distinct negative document whose normalized teacher score is closest to that anchor.
This ensures that the selected training samples structurally mimic the entire skeleton of the teacher's score distribution—capturing high, medium, and low relevance scores alike.
The authors evaluated this strategy against common baselines (retriever-top, reranker-top, mid, low, and random) using a two-stage training pipeline: an initial Contrastive Learning phase followed by Knowledge Distillation using either KL-Divergence or MarginMSE loss.
Datasets
- Training: MS MARCO-Train (8.8 million documents, 532,000 queries).
- In-Domain Evaluation: MS MARCO-Dev and TREC Deep Learning (DL) Track 19.
- Out-of-Domain Evaluation: The BEIR benchmark (evaluated across 13 diverse datasets to test zero-shot generalization).
Key Insights and Ideas
- Diversity Beats Pure Difficulty: Statistical metrics of the training data (like score coverage and entropy) directly correlate with downstream retrieval performance. Stratified sampling yielded the highest entropy and standard deviation, meaning it successfully preserved the teacher's nuanced landscape of preferences.
- The Danger of Regression Collapse: Regression-based distillation objectives like MarginMSE are incredibly sensitive to distributional bias. When trained exclusively on
reranker-tophard negatives using a Co-Condenser backbone, the model suffered from catastrophic training collapse (dropping to an MRR@10 of just 0.006). Using Stratified Sampling under the exact same conditions completely averted this, achieving a robust 0.307. - Robustness to Sample Size ($K$): As the number of negative samples ($K$) scales up, Stratified Sampling systematically fills in the gaps of the score range, providing complementary learning signals rather than redundant, highly correlated hard examples.
Key Results
Stratified Sampling consistently ranked at or near the top across multiple student backbones (BERT, DistilBERT, Co-Condenser) and both distillation objectives.
- In-Domain (MS MARCO Dev): With a Co-Condenser backbone and KL-Divergence, Stratified Sampling achieved an MRR@10 of 0.313, outperforming
reranker-top(0.255) andrandom(0.295). - Out-of-Domain (BEIR-13): Stratified Sampling demonstrated excellent zero-shot generalization, hitting an nDCG@10 of 0.376 (with Co-Condenser + MarginMSE), significantly beating out the standard
retriever-topbaseline (0.170).
Limitations and Future Work
To rigorously isolate the effect of score distributions, the authors intentionally restricted their experiments to a static, fixed candidate pool of 200 documents. While this controlled setup was necessary to prove their hypothesis, real-world production pipelines often utilize complex, dynamic negative miners that update throughout training.
The authors note that their work focuses purely on the static composition of data. A natural next step for future work would be exploring how to integrate the principles of Stratified Sampling into dynamic mining pipelines or combining it with complex curriculum learning schedules.
Summary updated