Tail-STEAK: Improve Friend Recommendation for Tail Users via Self-Training Enhanced Knowledge Distillation

Yijun Ma, Chaozhuo Li, Xiao Zhou

Introduction

In modern social networks, user connectivity typically follows a power-law distribution. While a small group of "head users" possesses hundreds of connections, the vast majority of "tail users" have very few. Current Graph Neural Network (GNN) based recommendation systems suffer from a significant "degree-related bias," performing poorly for these underserved tail users. This performance gap stems from two primary challenges: Label Sparsity, where few observed friendships exist to guide training, and Neighborhood Sparsity, where the lack of local structure makes it difficult for models to generalize preferences from well-connected users to sparse ones.

To address these issues, we present Tail-STEAK (Tail user oriented Self-Training EnhAnced Knowledge distillation), a framework designed to bridge the gap between head and tail user performance through iterative self-training and a novel knowledge distillation approach.

Methodology

Tail-STEAK operates through a two-stage process that combines self-training with self-knowledge distillation tasks to synthesize robust user representations.

Two-Stage Self-Training

To mitigate label sparsity, the framework first trains a base model exclusively on head users and their established connections. In the second stage, this model generates "pseudo-links" for tail users—predicting potential friendships based on learned patterns. These high-confidence predictions are then incorporated as additional training labels, effectively enriching the sparse data available for tail users.

Self-Knowledge Distillation and MI Maximization

To bridge the preference distribution gap, Tail-STEAK employs data augmentation to create synthetic views of users:

  • Head-to-Tail: Head users are transformed into synthetic tail users by aggressively dropping links and disturbing ID embeddings.
  • Tail-to-Head: Tail users are transformed into synthetic head users by imputing their neighborhoods with predicted pseudo-links.
The Tail-STEAK framework architecture, illustrating the two-stage self-training and mutual information (MI) maximization between head and tail views.

The framework utilizes Mutual Information (MI) Maximization to ensure consistency between these real and synthetic views. By forcing the model to align the distributions of a user's "head-like" and "tail-like" representations, it learns features that are invariant to the user's degree, leading to more robust recommendations.

Experimental Evaluation

The framework was evaluated on two real-world datasets: Deezer (28k nodes) and Last.FM (136k nodes). Both datasets exhibit strong power-law distributions. Tail-STEAK was implemented as a plug-and-play enhancement for two state-of-the-art GNN architectures: LightGCN and Simple-HGN (SHGN).

Results and Analysis

Experimental results demonstrate that Tail-STEAK significantly outperforms existing de-biasing and contrastive learning baselines, particularly for tail users.

Performance comparison of Tail-STEAK against various baselines across Deezer and Last.FM datasets, highlighting significant improvements for tail users (≤ T).

Key findings include:

  • Significant Tail Improvement: On the Last.FM dataset using LightGCN, Tail-STEAK improved the Mean Reciprocal Ranking (MRR) for tail users by 58.37% compared to the base model.
  • Preservation of Head Performance: Unlike many methods that improve tail performance at the expense of head users, Tail-STEAK maintains or even slightly improves performance for well-connected users.
  • Robustness of MI Maximization: The use of MI maximization proved superior to standard reconstruction losses, as it focuses on high-level distributional consistency rather than noisy feature-level details.

Conclusion

Tail-STEAK effectively addresses degree-related bias in social recommendations by synthesizing information across the head-tail divide. While the two-stage process adds some computational complexity and requires careful tuning of the degree threshold ($T$), its model-agnostic nature makes it a powerful tool for improving fairness and accuracy in sparse network environments. Future work may explore extending this approach to other link-prediction domains, such as knowledge graph completion.

Summary updated

Abstract

Graph neural networks (GNNs) are commonly employed in collaborative friend recommendation systems. Nevertheless, recent studies reveal a notable performance gap, particularly for users with limited connections, commonly known as tail users, in contrast to their counterparts with abundant connections (head users). Uniformly treating head and tail users poses two challenges for tail user preference learning: (C1) Label Sparsity, as tail users typically possess limited labels; and (C2) Neighborhood Sparsity, where tail users exhibit sparse observable friendships, leading to distinct preference distributions and performance degradation compared to head users. In response to these challenges, we introduce Tail-STEAK, an innovative framework that combines self-training with enhanced knowledge distillation for tail user representation learning. To address(C1), we present Tail-STEAK-base, a two-stage self-training framework. In the first stage, only head users and their accurate connections are utilized for training, while pseudo links are generated for tail users in the second stage. To tackle (C2), we propose two data augmentation-based self-knowledge distillation pretext tasks. These tasks are seamlessly integrated into different stages of Tail-STEAK-base, culminating in the comprehensive Tail-STEAK framework. Extensive experiments, conducted on state-of-the-art GNN-based friend recommendation models, substantiate the efficacy of Tail-STEAK in significantly improving tail user performance. Our code and data are publicly available at https://github.com/antman9914/Tail-STEAK.
BibTeX
@inproceedings{ma2024tail,
  author = {Yijun Ma and Chaozhuo Li and Xiao Zhou},
  title = {{Tail-STEAK: Improve Friend Recommendation for Tail Users via Self-Training Enhanced Knowledge Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i8.28737},
}
Figure