Bidirectional Distillation for Top-K Recommender System

Wonbin Kweon, SeongKu Kang, Hwanjo YuWWW 2021 · DOI · OpenReview

Abstract

Recommender systems (RS) have started to employ knowledge distillation (KD), a model compression technique training a compact student model with knowledge transferred from a cumbersome teacher model. State-of-the-art methods rely on unidirectional distillation, assuming the teacher is always superior. However, we demonstrate that the student performs better than the teacher on a significant proportion of the test set (36–42%). Based on this, we propose the Bidirectional Distillation (BD) framework, where both models collaboratively improve. We introduce a rank discrepancy-aware sampling scheme to distill informative knowledge effectively. Our experiments show that BD consistently outperforms competitors, with both teacher and student models showing significant improvements.

Introduction: Challenging the Superior Teacher Fallacy

Modern recommender systems are often massive, leading to high latency and memory requirements. Knowledge Distillation (KD) is typically used to compress these models. However, traditional KD assumes a unidirectional flow of knowledge from a "superior" teacher to a "subordinate" student.

The authors identify a critical flaw in this assumption for RS: the student often outperforms the teacher. Analysis of datasets like Foursquare and CiteULike reveals that the student ranks test items higher than the teacher in 36% to 42% of cases.

Rank difference on Foursquare showing student superiority in 42% of cases.

Furthermore, existing methods often distill items the teacher already ranks highly—information the student likely already possesses. This redundancy limits the effectiveness of the distillation process.

The Bidirectional Distillation (BD) Framework

To address these issues, the paper proposes Bidirectional Distillation (BD). Unlike standard KD, BD trains the teacher and student simultaneously, allowing for mutual knowledge exchange.

Overview of the Bidirectional Distillation (BD) framework architecture.

The core of the framework is the Rank Discrepancy-aware Sampling strategy. Instead of uniform sampling, BD focuses on items where the two models disagree most on the ranking:

  • Teacher → Student ($p_{T \to S}$): Uses a tanh function to provide "broad knowledge," sampling items the teacher ranks significantly higher than the student.
  • Student → Teacher ($p_{S \to T}$): Uses an exponential function to be highly selective, only transferring knowledge when the student is extremely confident about an item the teacher has missed.

Experimental Results

The framework was evaluated on real-world datasets including CiteULike, Foursquare, and Yelp, using base architectures like NeuMF, CDAE, and BPR.

Mutual Improvement

A key finding is that both models benefit from the interaction. The teacher's performance improved by up to 20.79%, while the student saw gains of up to 39.88% compared to standalone training. This suggests that the student captures simpler patterns that help regularize the teacher, while the teacher provides complex structural knowledge to the student.

Efficiency and Performance

BD-trained students consistently outperformed state-of-the-art distillation methods like Ranking Distillation (RD). Notably, a student model with only 10% of the teacher's parameters achieved performance comparable to or better than the original teacher on the Foursquare dataset.

Conclusion and Future Work

Bidirectional Distillation shifts the paradigm from "teacher-student" to "collaborative learning." By acknowledging that even smaller models have unique insights, BD achieves superior performance and efficiency. While simultaneous training increases initial computational overhead, the resulting performance gains and model compression make it a compelling approach for real-time recommendation. Future work may explore applying BD to explicit feedback and multi-behavior recommendation scenarios.

Summary updated

Abstract

Recommender systems (RS) have started to employ knowledge distillation, which is a model compression technique training a compact model (student) with the knowledge transferred from a cumbersome model (teacher). The state-of-the-art methods rely on unidirectional distillation transferring the knowledge only from the teacher to the student, with an underlying assumption that the teacher is always superior to the student. However, we demonstrate that the student performs better than the teacher on a significant proportion of the test set, especially for RS. Based on this observation, we propose Bidirectional Distillation (BD) framework whereby both the teacher and the student collaboratively improve with each other. Specifically, each model is trained with the distillation loss that makes to follow the other’s prediction along with its original loss function. For effective bidirectional distillation, we propose rank discrepancy-aware sampling scheme to distill only the informative knowledge that can fully enhance each other. The proposed scheme is designed to effectively cope with a large performance gap between the teacher and the student. Trained in the bidirectional way, it turns out that both the teacher and the student are significantly improved compared to when being trained separately. Our extensive experiments on real-world datasets show that our proposed framework consistently outperforms the state-of-the-art competitors. We also provide analyses for an in-depth understanding of BD and ablation studies to verify the effectiveness of each proposed component.
BibTeX
@inproceedings{kweon2021bidirectional,
  author = {Wonbin Kweon and SeongKu Kang and Hwanjo Yu},
  title = {{Bidirectional Distillation for Top-K Recommender System}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442381.3449878},
}
Figure