Introduction
Graph Neural Networks (GNNs) have become the standard for graph-based machine learning due to their ability to aggregate neighborhood information via message passing. However, this recursive neighborhood fetching introduces significant latency during inference, hindering real-time deployment. Multi-Layer Perceptrons (MLPs) offer a high-speed alternative by processing nodes independently, but they typically lack the structural awareness necessary for high performance on graph data.
Knowledge Distillation (KD) has emerged as a solution to bridge this gap, training a student MLP to mimic a teacher GNN. Yet, existing KD methods often rely on a simplistic "node-to-node" paradigm, aligning only final outputs and failing to capture the complex, multi-scale relationships between node pairs. This paper introduces MSN-GDM (Multi-Scale Node-pair grouping Graph-Distillation-MLP), a framework designed to transfer nuanced structural knowledge by grouping node pairs into multiple scales based on similarity.
Methodology
The MSN-GDM framework operates through three primary components: multi-scale grouping, tailored distillation losses, and hierarchical layer integration.
1. Multi-Scale Node Pair Grouping
Rather than treating all neighbors uniformly, MSN-GDM calculates a similarity metric, $\phi_k(i, j)$, for node pairs. This metric integrates feature similarity (attribute-level) and structural similarity (neighborhood-level). Node pairs are then categorized into $J$ distinct scale groups, ranging from highly similar to highly dissimilar.
2. Multi-Scale Distillation Loss
The model employs a dual-optimization strategy to capture both local and global structures:
- Intra-Layer Loss: Minimizes the distance between representations of neighboring nodes within a single layer, forcing the MLP to implicitly learn local graph topology.
- Inter-Layer Loss: This component applies different strategies based on the scale group. For similar node pairs, Aggregate Knowledge Distillation (AKD) pulls representations closer. For dissimilar pairs, Different Knowledge Distillation (DKD) is used to preserve their unique characteristics, preventing over-smoothing.
3. Hierarchical Weighted Sum
To account for the varying receptive fields captured by different GNN layers, the student MLP does not simply rely on its final layer. Instead, the final output is a learnable weighted sum of all internal layer representations, allowing the model to adaptively select the most relevant structural knowledge.
Experimental Results
The authors evaluated MSN-GDM across eight benchmarks, including citation networks (Cora, Pubmed), co-purchase networks (Amazon), and the large-scale Ogbn-Arxiv dataset.
- Accuracy: MSN-GDM consistently outperformed state-of-the-art methods like GLNN and FF-G2M. In several instances, the student MLP even surpassed the teacher GNN's performance (e.g., on the Citeseer dataset).
- Efficiency: The framework achieves massive inference speedups. On Cora, the MSN-GDM student recorded an inference time of 7.03ms, significantly faster than GCN (9.22ms) and BernNet (16.17ms). This advantage scales with graph size as the need for neighborhood fetching is eliminated.
- Parameter Economy: Student models often maintained or improved performance while utilizing fewer parameters than their GNN counterparts.
Discussion and Future Work
MSN-GDM demonstrates that MLPs can act as effective "structural mimics" if the distillation process accounts for the heterogeneity of node relationships. By explicitly separating similar and dissimilar node pairs, the model captures both low-frequency (smooth) and high-frequency (sharp) structural information.
Currently, the framework is primarily optimized for transductive settings where the graph structure is known during training. Future research will focus on extending this to inductive settings for unseen nodes and making the grouping scale ($J$) adaptive to specific graph topologies. Additionally, while inference is rapid, the preprocessing required for multi-scale grouping adds complexity to the training pipeline, which remains an area for optimization.
Summary updated