From Input to Output: A Multi-layer Knowledge Distillation Framework for Compressing Recommendation Models

Zhangchi Zhu, Wei ZhangarXiv 2023 · arXiv · OpenReview

TL;DR

This paper introduces Preference-Consistent Knowledge Distillation (PCKD), a novel approach to compress recommendation models. PCKD tackles the issue of "preference inconsistency" where projectors, used to align student and teacher model features, can distort user preferences, leading to wasted knowledge distillation potential. By introducing specific regularization terms for these projectors, PCKD significantly improves the performance of feature-based knowledge distillation, allowing smaller models to achieve comparable results to larger ones.

What Problem Does This Paper Address?

Modern recommendation systems often rely on large, complex models to accurately capture user preferences and item characteristics. While effective, these models are computationally expensive, leading to high storage costs and slow inference times, which negatively impact user experience. Knowledge distillation (KD) offers a solution by training smaller "student" models to mimic the behavior of larger "teacher" models.

A popular KD technique for recommenders is feature-based KD, which aims to transfer knowledge from the teacher's intermediate feature representations to the student's. This typically involves using "projectors" to align the dimensionality of student and teacher features, followed by minimizing the difference between them. However, the authors identify a critical flaw: these projectors, while aligning feature dimensions, can inadvertently distort user preferences. This phenomenon, termed preference inconsistency, means that even if the projected features correctly predict user preferences, the student's original features might not, indicating the student hasn't truly learned the underlying preference knowledge. This inconsistency significantly hinders the effectiveness of feature-based KD.

Methods

PCKD introduces two novel regularization terms designed to mitigate preference inconsistency:

  • Pair-wise PCKD: This term is inspired by pairwise ranking losses (like BPR) commonly used in recommenders. It aims to ensure that the preference relationship between two items (e.g., item A is preferred over item B) predicted by the student's features is consistent with the preference relationship predicted by the projected features.
  • List-wise PCKD: This term draws inspiration from listwise cross-entropy losses. It focuses on aligning the entire distribution of predicted preference scores for a list of items, ensuring that the projected features maintain the same preference distribution as the student's features.

Both methods employ rank-aware sampling to prioritize items with higher predicted preference scores, as these are crucial for effective knowledge distillation. The paper also proposes a hybrid PCKD (PCKD-H) that combines both pair-wise and list-wise terms for potentially better performance. PCKD is integrated into the overall training objective alongside the base recommendation model's loss and existing feature-based distillation losses.

Datasets

The authors evaluate their method on three public datasets:

  • CiteULike
  • Gowalla
  • Yelp2018

These datasets are standard benchmarks for recommendation system research. The paper does not introduce new datasets.

Key Insights and Ideas

  • Preference Inconsistency is a Widespread Problem: The paper empirically demonstrates that preference inconsistency is not a rare occurrence but a prevalent issue in feature-based KD for recommenders, even affecting items with high preference scores.
  • Projectors Can "Cheat": The core insight is that projectors, while necessary for dimensionality alignment, can act as a "cheat sheet" for the student model, making projected features appear to match the teacher's without the student truly learning the underlying user preferences.
  • Focusing on High-Preference Items is Crucial: The authors highlight that items with high preference scores are particularly important for knowledge distillation. Their rank-aware sampling strategy specifically targets these items to maximize the impact of PCKD.
  • Feature-Based KD Has Untapped Potential: The significant performance gains achieved by PCKD suggest that existing feature-based KD methods are underutilizing the knowledge embedded in teacher features due to the unaddressed preference inconsistency.

Key Results

  • Significant Performance Improvement: PCKD consistently outperforms existing response-based, feature-based, and relation-based knowledge distillation methods across all tested datasets and backbones.
  • Comparable to Teacher Models: In many cases, PCKD-trained student models achieve performance comparable to the original teacher models, demonstrating effective knowledge transfer.
  • Reduced Preference Inconsistency: Experiments show that PCKD significantly reduces preference inconsistency, particularly for items with similar preference scores and high-preference items.
  • Efficiency Gains: While PCKD adds some computational overhead compared to basic feature-based methods, it still offers substantial improvements in inference efficiency (lower latency, smaller model size) compared to the teacher model. List-wise PCKD offers better accuracy but at a higher training cost than pair-wise PCKD.

Limitations and Future Work

  • Training Cost Trade-off: The paper acknowledges that list-wise PCKD, while highly effective, incurs a larger training cost. This presents a trade-off for users who might need to prioritize training speed over marginal accuracy gains.
  • Future Work: The authors express a commitment to exploring ways to reduce preference inconsistency without increasing training costs, suggesting potential avenues for future research. They also investigate the application of PCKD to relation-based methods like HTD, showing its broad applicability.

Summary updated

Abstract

To reduce the size of recommendation models, there have been many studies on compressing recommendation models using knowledge distillation. In this paper, we decompose recommendation models into three layers, i.e., the input layer, the intermediate layer, and the output layer, and address deficiencies layer by layer. First, previous methods focus only on two layers, neglecting the input layer. Second, in the intermediate layer, existing methods ignore the inconsistency of user preferences induced by the projectors. Third, in the output layer, existing methods use only hard labels rather than soft labels from the teacher. To address these deficiencies, we propose \textbf{M}ulti-layer \textbf{K}nowledge \textbf{D}istillation (MKD), which consists of three components: 1) Distillation with Neighbor-based Knowledge (NKD) utilizes the teacher's knowledge about entities with similar characteristics in the input layer to enable the student to learn robust representations. 2) Distillation with Consistent Preference (CPD) reduces the inconsistency of user preferences caused by projectors in the intermediate layer by two regularization terms. 3) Distillation with Soft Labels (SLD) constructs soft labels in the output layer by considering the predictions of both the teacher and the student. Our extensive experiments show that MKD even outperforms the teacher with one-tenth of the model size.
BibTeX
@misc{zhu2023from,
  author = {Zhangchi Zhu and Wei Zhang},
  title = {{From Input to Output: A Multi-layer Knowledge Distillation Framework for Compressing Recommendation Models}},
  year = {2023},
  eprint = {2311.04549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.04549},
}

Loading related content...

Figure