TL;DR
When training modern language-model-based text rerankers, knowledge distillation (KD) from a larger teacher model consistently outperforms direct contrastive learning (CL) for smaller student models. However, distilling into a student of the same capacity as the teacher provides no benefit and can actively harm out-of-domain generalization.
What Problem Does This Paper Address?
Modern information retrieval systems typically use a two-stage pipeline: a fast initial retriever pulls a broad set of candidate documents, and a computationally heavier "reranker" re-orders them to maximize relevance. Today's state-of-the-art rerankers are cross-encoders built on top of decoder-only Large Language Models (LLMs).
To train these rerankers, developers generally choose between two strategies:
- Contrastive Learning (CL): Training the model directly on ground-truth labels to distinguish positive documents from hard negatives.
- Knowledge Distillation (KD): Training a smaller "student" model to mimic the exact output probability distribution (logits) of a larger, highly capable "teacher" model.
While both methods are widely used, there hasn't been a rigorous, apples-to-apples comparison of CL versus KD specifically for modern LLM-based cross-encoders. This paper bridges that gap, providing practical guidance on which strategy to choose based on your model size, architecture, and available compute.
Methods
The authors designed a highly controlled suite of experiments to isolate the impact of the training strategy. They trained cross-encoder rerankers across multiple sizes (0.5B, 1.5B, 3B, and 7B parameters) and two distinct architectures:
- Transformer-based: Qwen2.5 base models.
- Recurrent-based: RecurrentGemma (2B), which utilizes the linear-complexity Griffin architecture instead of standard quadratic self-attention.
For the contrastive learning baseline, models were optimized using the standard InfoNCE loss function. For knowledge distillation, they first trained a strong RankQwen-7B-CL model to act as the teacher. The student models were then trained by minimizing the Kullback-Leibler (KL) divergence between their softened output logits and the teacher's logits. Crucially, both CL and KD models were trained on the exact same query-document triples to ensure a completely fair comparison.
Datasets
- Training Data: The primary training set was constructed from the MS MARCO passage retrieval dataset (532k training pairs, 8.8M passages), with hard negatives mined using a custom-trained RepQwen retriever. To test data scaling, they also utilized RLHN, a multi-domain dataset containing 649k samples pruned from the BGE training mixture.
- Evaluation Benchmarks:
- In-domain: MS MARCO passage dev set, TREC DL19, and TREC DL20.
- Out-of-domain: 13 publicly available datasets from the BEIR benchmark (e.g., Arguana, Climate-FEVER, DBPedia, FiQA).
Key Insights and Ideas
- The "Soft Label" Advantage: KD outperforms CL for smaller models because the teacher's logits provide nuanced, continuous "soft labels" rather than binary 0/1 ground truths. This extra information acts as a regularizer, helping smaller models learn smoother decision boundaries.
- The Same-Capacity Distillation Trap: Distilling a 7B teacher into a 7B student yields no performance gains in-domain and causes a noticeable drop out-of-domain. When capacities match, the student overfits to the teacher's specific biases and quirks on the training set, degrading its ability to generalize to unseen domains.
- Architecture Agnostic Success: The superiority of KD over CL holds true not just for standard Transformers, but also for recurrent architectures like RecurrentGemma. This proves that the benefits of distillation are fundamental to the objective function, not the underlying model layers.
- Data Scaling Divergence: As the volume of training data scales up, CL eventually catches up to or slightly beats KD on in-domain tasks. However, KD maintains a robust and consistent advantage on out-of-domain datasets.
Key Results
- In-Domain Performance: On the MS MARCO Dev set, RankQwen-0.5B-KD achieved a 43.5 MRR@10, noticeably outperforming its contrastive counterpart, RankQwen-0.5B-CL, which scored 42.1.
- Out-of-Domain Generalization: On the BEIR benchmark, RankQwen-3B-KD achieved an average 57.6 NDCG@10, beating RankQwen-3B-CL (56.2) and even slightly outperforming the 7B teacher model itself (56.5).
The clear scaling trends and the performance gap between CL and KD across different model sizes can be seen in Figure 1:
When scaling the training data size (Figure 2), the authors observed that while in-domain performance converges between the two methods, KD retains a clear edge for out-of-domain tasks:
Limitations and Future Work
Due to computational constraints, the authors were unable to scale their experiments to models larger than 7B parameters (such as Qwen2.5-14B or 72B). It remains an open question whether these scaling trends hold true for double-digit billion-parameter models.
Additionally, this study focused entirely on off-policy distillation, where the student mimics static teacher outputs. The authors note that a promising future direction is exploring on-policy distillation (similar to reinforcement learning), where the student model generates its own candidate document rankings and receives dynamic feedback based on its current policy.
Summary updated