SALE-MLP: Structure Aware Latent Embeddings for GNN to Graph-free MLP Distillation

Harsh Pal, Sarthak Malik, Rajat Patel, Aakarsh MalhotraIJCAI 2025 · DOI

TL;DR

This paper introduces SALE-MLP, a novel method for distilling knowledge from Graph Neural Networks (GNNs) into efficient Multi-Layer Perceptrons (MLPs) without requiring the graph structure during inference. By incorporating unsupervised structural losses, SALE-MLP significantly improves MLP performance on node classification tasks, especially in challenging inductive settings, while maintaining graph-free efficiency.

What Problem Does This Paper Address?

Graph Neural Networks (GNNs) excel at modeling complex, non-Euclidean data like social networks or molecular structures, achieving state-of-the-art results in tasks like node classification. However, their inherent message-passing mechanism leads to high computational overhead and latency, making them impractical for real-time applications. Knowledge distillation (KD) offers a solution by transferring GNN knowledge to lightweight MLPs. Yet, existing GNN-to-MLP (G2M) distillation methods often fail to capture crucial graph structure, leading to suboptimal performance, especially in inductive settings where the model must generalize to unseen nodes or graphs. Furthermore, some methods still require access to the graph during inference, defeating the purpose of achieving graph-free efficiency. This paper aims to bridge this gap by developing a structure-aware G2M distillation method that is both effective and graph-free.

Methods

The core of SALE-MLP lies in its "Structure-Aware Latent Embeddings" approach. It consists of two main components:

  • Latent-Space Projector (LSP): This component maps node features into a latent space. Crucially, it leverages unsupervised structural losses, such as a DeepWalk-inspired loss, to ensure that nodes with similar graph structures are mapped to similar representations in this latent space. This is achieved by applying the SkipGram model (traditionally used for word embeddings) to the embedding space of the MLP, encouraging neighboring nodes in random walks to have close embeddings.
  • MLP Classifier: This standard MLP takes the latent embeddings generated by the LSP and makes the final predictions.

The overall training objective is a weighted combination of:

  1. Ground Truth Supervision (LGT): Standard cross-entropy loss using true labels.
  2. GNN Distillation Loss (LSL): KL-divergence loss between the MLP's predictions and the teacher GNN's soft labels, capturing the GNN's learned patterns.
  3. Unsupervised Structural Loss (LDWL): The DeepWalk-inspired loss that enforces structural awareness in the latent space.

This end-to-end training approach allows the MLP to learn structure-aware embeddings without explicit graph access during inference.

Datasets

The paper evaluates SALE-MLP on six widely-adopted public benchmark datasets:

  • Cora
  • Citeseer
  • Pubmed
  • Amazon-Photo
  • Amazon-Computer
  • ogbn-arxiv (a large-scale dataset)

These datasets span different scales and network types (citation networks and co-purchase networks) and are commonly used by existing G2M methods, allowing for fair comparison. Standard splits are used for Cora, Citeseer, and Pubmed, while specific splits are used for Amazon and ogbn-arxiv as per prior work.

Key Insights and Ideas

  • Structure is Key, Even for MLPs: The most significant insight is that even when distilling knowledge into MLPs (which are inherently graph-agnostic), explicitly encoding graph structure into their latent representations dramatically improves performance, especially in challenging inductive scenarios.
  • Unsupervised Structure for Free: The use of unsupervised structural losses (like DeepWalk) is a clever way to inject structural awareness without needing labeled graph data or complex graph-based operations during inference.
  • Inductive Performance Boost: SALE-MLP shows a particularly strong advantage in inductive settings, where traditional G2M methods often falter due to their lack of structural awareness. This highlights the practical relevance of the method for real-world applications where data is constantly evolving.
  • Balancing Supervision: The ablation studies reveal that while GNN distillation (LSL) is important, the unsupervised structural loss (LDWL) is critical for achieving the best performance, suggesting that learning from the graph's topology is as important as mimicking the teacher's output.

Key Results

  • Superior Accuracy: SALE-MLP consistently outperforms existing G2M methods (GLNN, NOSMOG, KRD, AdaGMLP) across all datasets and settings. It achieves a 3-4% improvement in node classification accuracy for inductive settings.
  • Efficiency: SALE-MLP achieves competitive accuracy with significantly reduced inference time, offering a 150x speedup over the teacher GNN (SAGE) while maintaining superior performance.
  • Structure-Awareness: The method demonstrates superior consistency with graph topology (measured by min-cut value) compared to other G2M methods.
  • Generalization: SALE-MLP shows strong generalization ability, performing well even with limited labeled data (low k-shot scenarios).
  • Robustness: The method is robust to heterophilic graph structures, outperforming other distillation methods on datasets like Actor and Wisconsin.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores hyperparameter sensitivity, finding the optimal balance (especially for $\lambda$ and $\alpha$) might still require careful tuning.
  • Teacher GNN Dependence: The performance of SALE-MLP is still dependent on the quality of the teacher GNN.
  • Future Work: The authors suggest exploring other unsupervised structural losses and investigating the impact of different teacher GNN architectures. The broad applicability and adaptability of SALE-MLP across various teachers and structural losses are highlighted as a key strength for future research.

Summary updated

BibTeX
@inproceedings{pal2025sale,
  author = {Harsh Pal and Sarthak Malik and Rajat Patel and Aakarsh Malhotra},
  title = {{SALE-MLP: Structure Aware Latent Embeddings for GNN to Graph-free MLP Distillation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/668},
}
Figure