RankDistil: Knowledge Distillation for Ranking

Sashank J. Reddi, Rama Kumar Pasumarthi, Aditya Krishna Menon, Ankit Singh Rawat, Felix X. Yu, Seungyeon Kim, Andreas Veit, Sanjiv KumarAISTATS 2021 · OpenReview

Abstract

Knowledge distillation is an approach to improve the performance of a student model by using the knowledge of a complex teacher. Despite its success in several deep learning applications, the study of distillation is mostly confined to classification settings. In particular, the use of distillation in top-k ranking settings, where the goal is to rank k most relevant items correctly, remains largely unexplored. In this paper, we study such ranking problems through the lens of distillation. We present a distillation framework for top-k ranking and draw connections with the existing ranking methods. The core idea of this framework is to preserve the ranking at the top by matching the order of items of student and teacher, while penalizing large scores for items ranked low by the teacher. Building on this, we develop a novel distillation approach, RankDistil, specifically catered towards ranking problems with a large number of items to rank, and establish statistical basis for the method. Finally, we conduct experiments which demonstrate that RankDistil yields benefits over commonly used baselines for ranking problems.

Introduction

Knowledge Distillation (KD) has become a cornerstone of modern AI efficiency, enabling compact "student" models to inherit the performance characteristics of massive "teacher" models. While KD is well-established for classification tasks, its application to ranking—a fundamental component of search engines and recommendation systems—presents unique challenges.

Ranking is inherently more complex than classification. Instead of predicting a single label, a model must order a list of items ($K$) for a specific query. Two primary hurdles complicate this:

  1. Scale: The candidate pool can contain millions of items, making traditional softmax-based distillation computationally prohibitive.
  2. Top-Heavy Importance: In ranking, the precision of the top-10 results is critical, whereas the relative order of items deep in the tail (e.g., rank 10,000 vs. 10,001) is practically irrelevant.

The RankDistil Framework

To address these challenges, the authors propose RankDistil, a framework centered on a $p$-focused distillation loss (p-FDL). The core philosophy is to ensure the student matches the teacher’s ordering for the top $p$ "positive" items while ensuring "negative" items (those ranked low by the teacher) remain below this top group.

Loss Families and Formalization

The framework unifies several ranking loss functions. As shown in the mathematical definitions below, the loss can be instantiated through various families, including coupled (list-wise), binary (point-wise), and pairwise approaches.

Mathematical definitions of the loss families used in the RankDistil framework.

Scalability through Negative Mining

To handle massive item sets, RankDistil employs negative sampling and mining. Rather than computing the loss over all $K$ items, the algorithm samples a small batch and identifies "hard" negatives—items the student incorrectly ranks highly. This strategy ensures training time remains independent of the total document pool size.

Experimental Evaluation

The researchers evaluated RankDistil across four major Learning to Rank (LTR) benchmarks: MSLR-WEB30K, Yahoo LETOR, Istella, and MSMARCO. The latter is a massive dataset from Bing queries, used to test scaling with BERT-based models.

Key Results

RankDistil consistently outperformed state-of-the-art distillation baselines across all datasets. Notably, it achieved significant gains in NDCG (Normalized Discounted Cumulative Gain) and MRR (Mean Reciprocal Rank).

Performance comparison of RankDistil against baselines on the MSMARCO and Istella datasets.

On the MSMARCO dataset, the authors demonstrated massive compression by distilling a BERT-Base teacher (110M parameters) into a BERT-Tiny student (4.4M parameters). The resulting student achieved a 96% reduction in size while significantly outperforming students trained only on raw ground-truth data. This suggests that the "smoothed" knowledge from a teacher is often more informative than sparse human labels like clicks.

Conclusion

RankDistil provides a statistically consistent and computationally efficient framework for distilling knowledge in top-$k$ ranking scenarios. By focusing on the top of the list and utilizing smart negative mining, it bridges the gap between massive teacher models and the low-latency requirements of production ranking systems. Future work may involve refining position-aware weights and extending the framework to cross-modal ranking tasks.

Summary updated

Abstract

Knowledge distillation is an approach to improve the performance of a student model by using the knowledge of a complex teacher. Despite its success in several deep learning applications, the study of distillation is mostly confined to classification settings. In particular, the use of distillation in top-k ranking settings, where the goal is to rank k most relevant items correctly, remains largely unexplored. In this paper, we study such ranking problems through the lens of distillation. We present a distillation framework for top-k ranking and draw connections with the existing ranking methods. The core idea of this framework is to preserve the ranking at the top by matching the order of items of student and teacher, while penalizing large scores for items ranked low by the teacher. Building on this, we develop a novel distillation approach, RankDistil, specifically catered towards ranking problems with a large number of items to rank, and establish statistical basis for the method. Finally, we conduct experiments which demonstrate that RankDistil yields benefits over commonly used baselines for ranking problems.
BibTeX
@inproceedings{reddi2021rankdistil,
  author = {Sashank J. Reddi and Rama Kumar Pasumarthi and Aditya Krishna Menon and Ankit Singh Rawat and Felix X. Yu and Seungyeon Kim and Andreas Veit and Sanjiv Kumar},
  title = {{RankDistil: Knowledge Distillation for Ranking}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=TzR4Q8XB-dv},
}
Figure