SHARP-Distill: A 68$\times$ Faster Recommender System with Hypergraph Neural Networks and Language Models

ICML 2025

TL;DR

This paper introduces SHARP-Distill, a novel knowledge distillation framework that significantly speeds up recommender systems by combining hypergraph neural networks (HGNNs) with language models. It achieves up to a 68x faster inference time compared to existing HGNN methods while maintaining competitive recommendation accuracy, making it ideal for real-world, large-scale applications.

What Problem Does This Paper Address?

Recommender systems are crucial for personalized user experiences, but many advanced models, particularly those using Graph Neural Networks (GNNs) and Hypergraph Neural Networks (HGNNs), suffer from high computational costs, leading to slow inference times. This makes them impractical for real-time applications. While knowledge distillation offers a way to create faster, lightweight models, existing methods struggle to effectively transfer the complex, high-order structural information learned by sophisticated teacher models, often sacrificing accuracy. This paper tackles the challenge of creating a recommender system that is both highly accurate and extremely fast, by effectively transferring knowledge from a powerful HGNN-based teacher to a lightweight student model.

Methods

SHARP-Distill employs a teacher-student knowledge distillation framework.

  • Teacher Model: This model is a sophisticated combination of:
  • Dual HGNNs: To capture complex, high-order relationships within user-item interactions.
  • DeBERTa (a language model): To extract rich semantic features from user reviews.
  • Contrastive Learning: A key innovation that aligns embeddings from different sources (user-item interactions and reviews) and ensures structural consistency between representations. This includes both cross-modal and intra-domain contrastive losses.
  • MLP: For predicting ratings.
  • Student Model: This is a lightweight model designed for efficiency:
  • CompactGCN: A shallow, single-layer GCN without non-linear activations. It's designed to inherit high-order relationships from the teacher while drastically reducing computational complexity.
  • Contrastive Learning: The student also uses contrastive learning, but crucially, it aligns its own embeddings with those from the teacher model. This alignment is enhanced by hypergraph positional encodings, which help transfer structural and positional knowledge.
  • Embedding Interpolation: A mechanism where the student's embeddings are interpolated with the teacher's, allowing for a controlled transition from teacher knowledge to student-learned representations during training.

The overall goal is to distill the rich structural and semantic knowledge from the complex teacher into the efficient student model.

Datasets

The paper evaluates SHARP-Distill on five well-known datasets:

  • Yelp Tips
  • Amazon Cellphones
  • Amazon Beauty
  • Amazon Sports
  • Amazon CDs

These datasets are chosen for their diversity in domain, size, and sparsity, and they all contain user interactions, items, and user reviews.

Key Insights and Ideas

  • Hypergraphs are powerful for high-order relationships: The paper leverages HGNNs to go beyond simple pairwise user-item interactions and capture more complex group dynamics.
  • Textual reviews add crucial semantic context: Integrating language models like DeBERTa significantly enhances recommendations by understanding user preferences and item characteristics from reviews.
  • Contrastive learning is key for effective knowledge transfer: Simply using soft labels isn't enough. Contrastive learning, especially with positional encodings, is vital for aligning diverse representations and preserving structural knowledge.
  • Shallow student models can be surprisingly effective: The CompactGCN, despite its simplicity, can inherit complex knowledge when guided by a sophisticated teacher and contrastive learning.
  • Efficiency doesn't have to mean sacrificing accuracy: SHARP-Distill demonstrates that it's possible to achieve massive speedups without a significant drop in recommendation quality.

Key Results

  • Inference Speed: SHARP-Distill achieves up to 68x faster inference compared to the HGNN teacher model and 40x faster than LightGCN.
  • Accuracy: The model maintains competitive recommendation accuracy, often surpassing baselines in metrics like Hit Ratio (HR@10, HR@20, HR@50), Precision@10 (P@10), Recall@10 (R@10), and NDCG@10 (NDCG@10). For example, it achieves the highest HR on 5 out of 6 metric-dataset combinations for Yelp and CDs.
  • Performance-Efficiency Trade-off: SHARP-Distill consistently occupies the optimal region in performance-efficiency plots, offering high accuracy with very low latency (sub-8ms inference time).
  • Component Contributions: Ablation studies show that DeBERTa contributes 18.81% to performance, HGNN contributes 24.48%, and contrastive learning is the most critical component, with a 29.38% relative drop when removed.
  • Scalability: The model demonstrates consistent performance and efficiency gains across datasets of varying sizes, from small-scale to large-scale.

Limitations and Future Work

  • Teacher Model Training Cost: While the student model is highly efficient for inference, the teacher model's training is computationally intensive. However, this is a one-time offline cost.
  • Hyperparameter Tuning: The performance is sensitive to hyperparameters like temperature for contrastive learning, contrastive loss weight, and distillation loss weight. Careful tuning is required.
  • Future Work: The authors suggest incorporating fairness constraints, diversity metrics, and transparency mechanisms. They also highlight the need for future work to explicitly address algorithmic fairness and develop techniques for detecting and mitigating potential harms in efficient recommendation systems. The paper also mentions further evaluations of DeBERTa embedding dimensions, traditional word embeddings, and the impact of disentangled attention.

Summary updated

BibTeX
@inproceedings{2025sharp,
  title = {{SHARP-Distill: A 68$\textbackslash\{\}times$ Faster Recommender System with Hypergraph Neural Networks and Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
}
Figure