Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks

Chenxiao Yang, Qitian Wu, Junchi Yan

Abstract

We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on a complete graph to a student GNN model operating on a smaller or sparser graph. To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs. A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as Geometric Knowledge Distillation. We develop non- and parametric instantiations and demonstrate their efficacy in various experimental settings for knowledge distillation regarding different types of privileged topological information and teacher-student schemes.

Introduction

Graph Neural Networks (GNNs) have revolutionized the processing of relational data, yet they suffer from a fundamental "topology dependency." Their performance is intrinsically linked to the quality and completeness of the input graph. In practical applications, however, graphs are frequently noisy, incomplete, or too massive to process in their entirety. This creates a gap between the theoretical potential of GNNs and their real-world deployment.

This research introduces Geometric Knowledge Distillation (GKD), a framework designed to encode graph topology as a "geometric prior" directly into a GNN's weights. By doing so, a "student" GNN can operate on a sparse or partial graph while maintaining the performance of a "teacher" model trained on a complete, privileged graph. This process, termed Geometric Knowledge Transfer, effectively compresses complex topological information into the model's parameters.

Methodology: The Neural Heat Kernel

The core innovation of GKD is the Neural Heat Kernel (NHK), which bridges the gap between graph topology and neural architecture through a thermodynamic lens.

1. The Thermodynamic Analogy

The authors revisit the relationship between GNNs and thermodynamics. Just as heat diffuses across a physical surface according to the geometry of the manifold, information "diffuses" across a graph during the message-passing phases of a GNN. The Heat Kernel is the mathematical operator that describes this flow.

2. Defining the NHK

The NHK is defined to encapsulate how a specific GNN architecture and its learned parameters govern the flow of information between node pairs. The researchers provide a rigorous proof for the existence of these kernels within standard GNN architectures, treating the graph as a discretization of an underlying geometric manifold.

3. The GKD Framework

To transfer knowledge, the student model is trained to align its NHK with that of the teacher. This alignment ensures that the student propagates features across its sparse graph in a manner that mimics the teacher’s propagation on the full graph. This goes beyond traditional Knowledge Distillation (KD), which typically only matches final output predictions; GKD forces the student to adopt the internal relational logic of the teacher.

4. Instantiations

  • Non-parametric GKD: Utilizes fixed kernels (such as Gaussian or Sigmoid) or randomized kernels based on graph eigenfunctions to approximate the NHK.
  • Parametric GKD (PGKD): Employs a variational approach where an independent "inverse-NHK" is learned to reconstruct features, allowing the model to learn the geometry in a data-driven, end-to-end fashion.

Experimental Evaluation

The researchers evaluated GKD across several standard benchmarks and experimental settings:

  • Datasets: Citation networks (Cora, Citeseer, PubMed) and the large-scale OGBN-Arxiv benchmark.
  • Edge-aware Scenarios: The student model is provided with significantly fewer edges than the teacher.
  • Node-aware Scenarios: The student model is provided with fewer labeled nodes.

Key Results

  • Superior Accuracy: On the Cora dataset in edge-aware settings, GKD variants achieved 88-89% accuracy. This significantly outperformed standard KD (84.8%) and nearly matched the "Oracle" model (88.6%)—a theoretical upper bound with access to the full graph during testing.
  • Robustness to Sparsity: While standard GNNs typically show sharp performance degradation as edges are removed, GKD-trained models maintain high accuracy, demonstrating that the topological prior is successfully embedded in the weights.
  • Efficiency and Scalability: Student models act as "topology-compressed" versions of the teacher. They achieve high performance with fewer edges, leading to faster inference. Furthermore, the authors utilized mini-batching techniques to scale the $O(n^2)$ kernel computations to large-scale graphs like OGBN-Arxiv.

Discussion and Insights

GKD represents a shift in how we view graph structure—not just as an input, but as a latent manifold that can be distilled. By matching the internal relational logic of the teacher, the student model becomes less dependent on the physical presence of edges during inference.

However, challenges remain. The computational overhead of computing node-pair relationships and the $O(n^2)$ space complexity for kernel matrices can be demanding for extremely dense graphs. Additionally, the parametric version (PGKD) requires an iterative training process that may take longer to converge than non-parametric alternatives.

Conclusion

Geometric Knowledge Distillation provides a principled method for transferring topological knowledge between GNNs. By leveraging the Neural Heat Kernel, GKD allows models to overcome the limitations of sparse or noisy data, paving the way for more robust and efficient graph representation learning in resource-constrained environments.

Summary updated

Abstract

We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on a complete graph to a student GNN model operating on a smaller or sparser graph. To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs. A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as Geometric Knowledge Distillation. We develop non- and parametric instantiations and demonstrate their efficacy in various experimental settings for knowledge distillation regarding different types of privileged topological information and teacher-student schemes.
BibTeX
@inproceedings{yang2022geometric,
  author = {Chenxiao Yang and Qitian Wu and Junchi Yan},
  title = {{Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=BsrKpkQzbM},
}
Figure