Abstract
Recommender systems (RS) have started to employ knowledge distillation (KD), a model compression technique training a compact student model with knowledge transferred from a cumbersome teacher model. State-of-the-art methods rely on unidirectional distillation, assuming the teacher is always superior. However, we demonstrate that the student performs better than the teacher on a significant proportion of the test set (36–42%). Based on this, we propose the Bidirectional Distillation (BD) framework, where both models collaboratively improve. We introduce a rank discrepancy-aware sampling scheme to distill informative knowledge effectively. Our experiments show that BD consistently outperforms competitors, with both teacher and student models showing significant improvements.
Introduction: Challenging the Superior Teacher Fallacy
Modern recommender systems are often massive, leading to high latency and memory requirements. Knowledge Distillation (KD) is typically used to compress these models. However, traditional KD assumes a unidirectional flow of knowledge from a "superior" teacher to a "subordinate" student.
The authors identify a critical flaw in this assumption for RS: the student often outperforms the teacher. Analysis of datasets like Foursquare and CiteULike reveals that the student ranks test items higher than the teacher in 36% to 42% of cases.
Furthermore, existing methods often distill items the teacher already ranks highly—information the student likely already possesses. This redundancy limits the effectiveness of the distillation process.
The Bidirectional Distillation (BD) Framework
To address these issues, the paper proposes Bidirectional Distillation (BD). Unlike standard KD, BD trains the teacher and student simultaneously, allowing for mutual knowledge exchange.
The core of the framework is the Rank Discrepancy-aware Sampling strategy. Instead of uniform sampling, BD focuses on items where the two models disagree most on the ranking:
- Teacher → Student ($p_{T \to S}$): Uses a
tanhfunction to provide "broad knowledge," sampling items the teacher ranks significantly higher than the student. - Student → Teacher ($p_{S \to T}$): Uses an
exponentialfunction to be highly selective, only transferring knowledge when the student is extremely confident about an item the teacher has missed.
Experimental Results
The framework was evaluated on real-world datasets including CiteULike, Foursquare, and Yelp, using base architectures like NeuMF, CDAE, and BPR.
Mutual Improvement
A key finding is that both models benefit from the interaction. The teacher's performance improved by up to 20.79%, while the student saw gains of up to 39.88% compared to standalone training. This suggests that the student captures simpler patterns that help regularize the teacher, while the teacher provides complex structural knowledge to the student.
Efficiency and Performance
BD-trained students consistently outperformed state-of-the-art distillation methods like Ranking Distillation (RD). Notably, a student model with only 10% of the teacher's parameters achieved performance comparable to or better than the original teacher on the Foursquare dataset.
Conclusion and Future Work
Bidirectional Distillation shifts the paradigm from "teacher-student" to "collaborative learning." By acknowledging that even smaller models have unique insights, BD achieves superior performance and efficiency. While simultaneous training increases initial computational overhead, the resulting performance gains and model compression make it a compelling approach for real-time recommendation. Future work may explore applying BD to explicit feedback and multi-behavior recommendation scenarios.
Summary updated