Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval

Youngjoon Jang, Seongtae Hong, Hyeonseok Moon, Heuiseok LimarXiv 2026 · arXiv

TL;DR

When training dense retrieval models via knowledge distillation, the common wisdom is to aggressively mine "hard negatives." This paper challenges that paradigm, demonstrating that focusing exclusively on hard negatives blinds the student model to the teacher's broader preference structure. By introducing a simple, parameter-free Stratified Sampling strategy that uniformly covers the teacher's entire score spectrum, the authors achieve superior generalization across both in-domain and out-of-domain retrieval benchmarks.

What Problem Does This Paper Address?

In modern Information Retrieval (IR), we often use a two-stage approach: a fast but less accurate dense retriever (bi-encoder) narrows down millions of documents to a few hundred, and a powerful but slow reranker (cross-encoder) scores them precisely. To make the dense retriever smarter, researchers use Knowledge Distillation (KD), where the dense retriever (the student) learns to mimic the continuous score outputs of the reranker (the teacher).

However, how do we choose which negative documents to use during this distillation training? The standard practice relies heavily on heuristics, such as picking the top-$K$ hardest negatives found by an initial retriever. The authors argue that this approach provides a highly biased, narrow view of the teacher's knowledge. If the student only sees extremely difficult negatives, it struggles to map out robust decision boundaries for varying levels of difficulty, ultimately hurting its ability to generalize to new, unseen domains.

Methods

To isolate the impact of score distributions from the messy variables of dynamic data mining, the authors created a controlled candidate pool of 200 negative documents per query (100 top-retrieved documents + 100 randomly sampled documents).

Instead of just picking the top-scoring documents from this pool, they proposed Stratified Sampling. The process is straightforward and deterministic:

  1. Normalize Scores: Apply query-level min-max normalization to the teacher's scores so they span from 0 to 1.
  2. Set Quantile Anchors: Evenly space out $K$ target probabilities (anchors) across the distribution.
  3. Select Closest Matches: For each anchor, select the distinct negative document whose normalized teacher score is closest to that anchor.

This ensures that the selected training samples structurally mimic the entire skeleton of the teacher's score distribution—capturing high, medium, and low relevance scores alike.

Figure 1: Illustration of candidate selection across different sampling strategies based on the teacher’s score distribution.

The authors evaluated this strategy against common baselines (retriever-top, reranker-top, mid, low, and random) using a two-stage training pipeline: an initial Contrastive Learning phase followed by Knowledge Distillation using either KL-Divergence or MarginMSE loss.

Datasets

  • Training: MS MARCO-Train (8.8 million documents, 532,000 queries).
  • In-Domain Evaluation: MS MARCO-Dev and TREC Deep Learning (DL) Track 19.
  • Out-of-Domain Evaluation: The BEIR benchmark (evaluated across 13 diverse datasets to test zero-shot generalization).

Key Insights and Ideas

  • Diversity Beats Pure Difficulty: Statistical metrics of the training data (like score coverage and entropy) directly correlate with downstream retrieval performance. Stratified sampling yielded the highest entropy and standard deviation, meaning it successfully preserved the teacher's nuanced landscape of preferences.
  • The Danger of Regression Collapse: Regression-based distillation objectives like MarginMSE are incredibly sensitive to distributional bias. When trained exclusively on reranker-top hard negatives using a Co-Condenser backbone, the model suffered from catastrophic training collapse (dropping to an MRR@10 of just 0.006). Using Stratified Sampling under the exact same conditions completely averted this, achieving a robust 0.307.
  • Robustness to Sample Size ($K$): As the number of negative samples ($K$) scales up, Stratified Sampling systematically fills in the gaps of the score range, providing complementary learning signals rather than redundant, highly correlated hard examples.

Key Results

Stratified Sampling consistently ranked at or near the top across multiple student backbones (BERT, DistilBERT, Co-Condenser) and both distillation objectives.

  • In-Domain (MS MARCO Dev): With a Co-Condenser backbone and KL-Divergence, Stratified Sampling achieved an MRR@10 of 0.313, outperforming reranker-top (0.255) and random (0.295).
  • Out-of-Domain (BEIR-13): Stratified Sampling demonstrated excellent zero-shot generalization, hitting an nDCG@10 of 0.376 (with Co-Condenser + MarginMSE), significantly beating out the standard retriever-top baseline (0.170).
Retrieval performance (nDCG@10) on TREC DL 19 (in-domain) and BEIR (out-of-domain) as the number of sampled candidates K varies.

Limitations and Future Work

To rigorously isolate the effect of score distributions, the authors intentionally restricted their experiments to a static, fixed candidate pool of 200 documents. While this controlled setup was necessary to prove their hypothesis, real-world production pipelines often utilize complex, dynamic negative miners that update throughout training.

The authors note that their work focuses purely on the static composition of data. A natural next step for future work would be exploring how to integrate the principles of Stratified Sampling into dynamic mining pipelines or combining it with complex curriculum learning schedules.

Summary updated

Abstract

Transferring knowledge from a cross-encoder teacher via Knowledge Distillation (KD) has become a standard paradigm for training retrieval models. While existing studies have largely focused on mining hard negatives to improve discrimination, the systematic composition of training data and the resulting teacher score distribution have received relatively less attention. In this work, we highlight that focusing solely on hard negatives prevents the student from learning the comprehensive preference structure of the teacher, potentially hampering generalization. To effectively emulate the teacher score distribution, we propose a Stratified Sampling strategy that uniformly covers the entire score spectrum. Experiments on in-domain and out-of-domain benchmarks confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline, significantly outperforming top-K and random sampling in diverse settings. These findings suggest that the essence of distillation lies in preserving the diverse range of relative scores perceived by the teacher.
BibTeX
@misc{jang2026beyond,
  author = {Youngjoon Jang and Seongtae Hong and Hyeonseok Moon and Heuiseok Lim},
  title = {{Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval}},
  year = {2026},
  eprint = {2604.04734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.04734},
}
Figure