Distillation versus Contrastive Learning: How to Train Your Rerankers

Zhichao Xu, Zhiqi Huang, Shengyao Zhuang, Vivek Srikumar

TL;DR

When training modern language-model-based text rerankers, knowledge distillation (KD) from a larger teacher model consistently outperforms direct contrastive learning (CL) for smaller student models. However, distilling into a student of the same capacity as the teacher provides no benefit and can actively harm out-of-domain generalization.

What Problem Does This Paper Address?

Modern information retrieval systems typically use a two-stage pipeline: a fast initial retriever pulls a broad set of candidate documents, and a computationally heavier "reranker" re-orders them to maximize relevance. Today's state-of-the-art rerankers are cross-encoders built on top of decoder-only Large Language Models (LLMs).

To train these rerankers, developers generally choose between two strategies:

  1. Contrastive Learning (CL): Training the model directly on ground-truth labels to distinguish positive documents from hard negatives.
  2. Knowledge Distillation (KD): Training a smaller "student" model to mimic the exact output probability distribution (logits) of a larger, highly capable "teacher" model.

While both methods are widely used, there hasn't been a rigorous, apples-to-apples comparison of CL versus KD specifically for modern LLM-based cross-encoders. This paper bridges that gap, providing practical guidance on which strategy to choose based on your model size, architecture, and available compute.

Methods

The authors designed a highly controlled suite of experiments to isolate the impact of the training strategy. They trained cross-encoder rerankers across multiple sizes (0.5B, 1.5B, 3B, and 7B parameters) and two distinct architectures:

  • Transformer-based: Qwen2.5 base models.
  • Recurrent-based: RecurrentGemma (2B), which utilizes the linear-complexity Griffin architecture instead of standard quadratic self-attention.

For the contrastive learning baseline, models were optimized using the standard InfoNCE loss function. For knowledge distillation, they first trained a strong RankQwen-7B-CL model to act as the teacher. The student models were then trained by minimizing the Kullback-Leibler (KL) divergence between their softened output logits and the teacher's logits. Crucially, both CL and KD models were trained on the exact same query-document triples to ensure a completely fair comparison.

Datasets

  • Training Data: The primary training set was constructed from the MS MARCO passage retrieval dataset (532k training pairs, 8.8M passages), with hard negatives mined using a custom-trained RepQwen retriever. To test data scaling, they also utilized RLHN, a multi-domain dataset containing 649k samples pruned from the BGE training mixture.
  • Evaluation Benchmarks:
  • In-domain: MS MARCO passage dev set, TREC DL19, and TREC DL20.
  • Out-of-domain: 13 publicly available datasets from the BEIR benchmark (e.g., Arguana, Climate-FEVER, DBPedia, FiQA).

Key Insights and Ideas

  • The "Soft Label" Advantage: KD outperforms CL for smaller models because the teacher's logits provide nuanced, continuous "soft labels" rather than binary 0/1 ground truths. This extra information acts as a regularizer, helping smaller models learn smoother decision boundaries.
  • The Same-Capacity Distillation Trap: Distilling a 7B teacher into a 7B student yields no performance gains in-domain and causes a noticeable drop out-of-domain. When capacities match, the student overfits to the teacher's specific biases and quirks on the training set, degrading its ability to generalize to unseen domains.
  • Architecture Agnostic Success: The superiority of KD over CL holds true not just for standard Transformers, but also for recurrent architectures like RecurrentGemma. This proves that the benefits of distillation are fundamental to the objective function, not the underlying model layers.
  • Data Scaling Divergence: As the volume of training data scales up, CL eventually catches up to or slightly beats KD on in-domain tasks. However, KD maintains a robust and consistent advantage on out-of-domain datasets.

Key Results

  • In-Domain Performance: On the MS MARCO Dev set, RankQwen-0.5B-KD achieved a 43.5 MRR@10, noticeably outperforming its contrastive counterpart, RankQwen-0.5B-CL, which scored 42.1.
  • Out-of-Domain Generalization: On the BEIR benchmark, RankQwen-3B-KD achieved an average 57.6 NDCG@10, beating RankQwen-3B-CL (56.2) and even slightly outperforming the 7B teacher model itself (56.5).

The clear scaling trends and the performance gap between CL and KD across different model sizes can be seen in Figure 1:

Figure 1: Scaling trend for contrastive learning vs knowledge distillation, for passage reranking. Left figure shows TREC DL19+20 results while right figure shows BEIR results averaged over 12 datasets (excluding Quora). We skip results of RankQwen-7B-KD model.

When scaling the training data size (Figure 2), the authors observed that while in-domain performance converges between the two methods, KD retains a clear edge for out-of-domain tasks:

Figure 2: Dataset scaling trend for CL versus KD. Left figure shows average results for 7 in-domain datasets from BEIR benchmark, while the right figure shows average results for 7 out-of-domain datasets.

Limitations and Future Work

Due to computational constraints, the authors were unable to scale their experiments to models larger than 7B parameters (such as Qwen2.5-14B or 72B). It remains an open question whether these scaling trends hold true for double-digit billion-parameter models.

Additionally, this study focused entirely on off-policy distillation, where the student mimics static teacher outputs. The authors note that a promising future direction is exploring on-policy distillation (similar to reinforcement learning), where the student model generates its own candidate document rankings and receives dynamic feedback based on its current policy.

Summary updated

Abstract

Training effective text rerankers is crucial for information retrieval. Two strategies are widely used: contrastive learning (optimizing directly on ground-truth labels) and knowledge distillation (transferring knowledge from a larger reranker). While both have been studied extensively, a clear comparison of their effectiveness for training cross-encoder rerankers under practical conditions is needed. This paper empirically compares these strategies by training rerankers of different sizes (0.5B, 1.5B, 3B, 7B) and architectures (Transformer, Recurrent) using both methods on the same data, with a strong contrastive learning model acting as the distillation teacher. Our results show that knowledge distillation generally yields better in-domain and out-of-domain ranking performance than contrastive learning when distilling from a more performant teacher model. This finding is consistent across student model sizes and architectures. However, distilling from a teacher of the same capacity does not provide the same advantage, particularly for out-of-domain tasks. These findings offer practical guidance for choosing a training strategy based on available teacher models. We recommend using knowledge distillation to train smaller rerankers if a larger, more performant teacher is accessible; in its absence, contrastive learning remains a robust baseline. Our code implementation is made available to facilitate reproducbility.
BibTeX
@misc{xu2025distillation,
  author = {Zhichao Xu and Zhiqi Huang and Shengyao Zhuang and Vivek Srikumar},
  title = {{Distillation versus Contrastive Learning: How to Train Your Rerankers}},
  year = {2025},
  eprint = {2507.08336},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.08336},
}
Figure