Introduction: The Depth-Degree Dilemma in GNNs
Graph Neural Networks (GNNs) face a significant challenge: the "one size fits all" approach to model depth often fails due to the heterogeneous nature of graphs. The performance of a GNN is intrinsically linked to a node's degree—the number of its connections.
As illustrated in Figure 1, high-degree nodes in dense neighborhoods require deeper architectures to capture wider contextual information. Conversely, low-degree nodes in sparse regions suffer from over-smoothing when processed by deep models, where repeated message passing makes node features indistinguishable. Standard Knowledge Distillation (KD) typically employs a single teacher, which cannot be optimal for every node scale.
Multi-Scale Knowledge Distillation (MSKD)
To address this sub-optimality, the authors propose Multi-Scale Knowledge Distillation (MSKD). This framework distills knowledge from an ensemble of teachers with varying depths, allowing a single compact student model to excel across diverse local structures.
The MSKD framework consists of three core components:
- Multi-layer GNN Teachers: A set of pre-trained teachers with different depths (e.g., 1, 2, and 3 layers) covers the spectrum of topological scales.
- Topological Semantic Mapping: Instead of simple label mimicry, the student learns to replicate the local structure by matching the similarity distribution between a center node and its neighbors using Kullback-Leibler (KL) divergence.
- Attentional Amalgamation: An attention mechanism dynamically weights teacher contributions for each node. High-degree nodes lean on deeper teachers, while low-degree nodes prioritize shallower ones, effectively "amalgamating" expertise from the entire teacher pool.
Experimental Evaluation
The researchers evaluated MSKD on four standard benchmarks: PPI (Protein-Protein Interaction), characterized by high average degrees, and Cora, CiteSeer, and PubMed, which feature lower average degrees.
Key Results
- Performance Gains: MSKD consistently outperformed state-of-the-art distillation methods like LSP, FitNet, and AT. On the PPI dataset using GAT, MSKD achieved a 97.31% F1-score, surpassing even the best individual teacher in the ensemble.
- Model Compression: The resulting student models are significantly more compact than the teacher ensemble while maintaining or improving accuracy.
- Feature Representation: t-SNE visualizations demonstrate that MSKD produces tighter, more distinct clusters compared to previous methods, indicating superior feature learning and semantic separation.
Conclusion and Future Directions
MSKD demonstrates that selective learning through attention allows a student model to overcome the inherent limitations of fixed-depth GNNs. By leveraging the depth-degree correlation, MSKD avoids both over-smoothing and under-fitting. Future work may explore automated teacher selection and extending the framework to dynamic or more complex graph architectures.
Summary updated