Deep Graph Mating

Yongcheng Jing, Seok-Hee Hong, Dacheng Tao

TL;DR

Researchers have introduced Deep Graph Mating (GRAMA), the first framework for merging pre-trained Graph Neural Networks (GNNs) without any re-training, fine-tuning, or human-annotated labels. By using a novel "Dual-Message Coordination and Calibration" (DuMCC) strategy, they can create a "child" model that inherits the expertise of its "parents" while remaining entirely learning-free.

The Challenge of GNN Model Reuse

In the world of deep learning, "model reuse" is a highly sought-after capability. Instead of training a massive model from scratch, it is often vastly more efficient to combine existing ones. While this works reasonably well for images (Euclidean data) using simple weight averaging, it fails miserably for graphs.

Comparison of graph-centric model reuse tasks. GRAMA is uniquely training-free and annotation-free.

Graph Neural Networks are notoriously sensitive to how neurons are ordered (permutation invariance) and the specific topology of the graphs they process. If you simply average the weights of two GNNs trained on different datasets, the resulting model usually performs poorly because the internal "features" aren't aligned. Existing solutions for GNN reuse, like Knowledge Distillation or Knowledge Amalgamation, require resource-intensive re-training or annotations, which defeats the purpose of a quick, efficient merge.

The DuMCC Methodology

To solve this, the authors propose DuMCC (Dual-Message Coordination and Calibration), a two-step methodology to "mate" parent GNNs:

  1. Parent Message Coordination (PMC): Instead of just looking at raw weights, PMC looks at the messages (aggregated neighbor data) the parent models produce. It finds the optimal way to permute (reorder) the neurons of one parent to match the other by ensuring their output messages are as similar as possible. This makes the alignment "topology-aware."
  2. Child Message Calibration (CMC): The authors discovered that even with perfect alignment, the "child" model suffers from over-smoothing. Because the child's features are essentially an average of the parents, the variance of its internal signals drops, making it less expressive. To fix this, they introduce a Learning-Free Message Normalization (LFNorm) layer. This layer uses the statistical mean and variance from the parent models to "rescale" the child's messages, restoring their discriminative power without needing a single gradient update.

Key Insights

  • GNNs are "Fragile" to Misalignment: The paper mathematically demonstrates that GNNs are much more sensitive to parameter mismatches than standard CNNs. A tiny misalignment in a GNN is amplified by the connectivity of the graph.
  • The "Averaging" Trap: Simply interpolating two models leads to a "variance reduction" effect. In GNNs, this manifests as over-smoothing, where node features become too similar to one another, losing the unique characteristics needed for classification.
  • Better than Training? Surprisingly, in some multi-label tasks, this training-free "mating" outperformed Knowledge Amalgamation (KA)—a method that actually involves re-training. This is because KA relies on "soft labels" from teachers which can contain errors; GRAMA avoids these errors by directly merging the learned logic of the parents.

Experimental Results

The framework was evaluated across seven diverse benchmarks covering various graph-related tasks, including node property prediction (ogbn-arxiv, ogbn-products), 3D object recognition (ModelNet40), large-scale semantic parsing (S3DIS), and molecular property prediction (ogbg-molbace). The experiments spanned five different GNN architectures (GCN, GraphSAGE, GAT, GIN, and DGCNN).

Performance comparison on ogbn-arxiv and ogbn-products datasets. DuMCC (Ours w/ CMC) significantly outperforms vanilla interpolation methods without retraining.
  • Training-Free Success: Across all benchmarks, DuMCC achieved results on par with (and sometimes better than) the original pre-trained parent models on their respective tasks.
  • Massive Improvement over Vanilla Merging: In 3D semantic segmentation (S3DIS), vanilla weight interpolation (VPI) yielded a mean accuracy of only 29.25%. The proposed DuMCC boosted this to 45.44%, nearing the performance of models that were actually trained on the data.
  • Efficiency: Because it requires no backpropagation, the "mating" process is near-instantaneous compared to traditional distillation or fine-tuning.

Limitations and Future Work

The authors acknowledge a few key constraints. Currently, the "parents" must have the same architecture (e.g., you cannot mate a GCN with a GAT), and they must address tasks at the same structural level (e.g., both node-level or both graph-level). Future work will investigate heterogeneous GRAMA to allow for merging different architectures, and explore how this technique could serve as a "warm-start" for more complex knowledge amalgamation tasks.

Summary updated

Abstract

In this paper, we introduce the first learning-free model reuse task within the non-Euclidean domain, termed as Deep Graph Mating (Grama). We strive to create a child Graph Neural Network (GNN) that integrates knowledge from pre-trained parent models without requiring re-training, fine-tuning, or annotated labels. To this end, we begin by investigating the permutation invariance property of GNNs, which leads us to develop two vanilla approaches for Grama: Vanilla Parameter Interpolation (VPI) and Vanilla Alignment Prior to Interpolation (VAPI), both employing topology-independent interpolation in the parameter space. However, neither approach has achieved the anticipated results. Through theoretical analysis of VPI and VAPI, we identify critical challenges unique to Grama, including increased sensitivity to parameter misalignment and further the inherent topology-dependent complexities. Motivated by these findings, we propose the Dual-Message Coordination and Calibration (DuMCC) methodology, comprising the Parent Message Coordination (PMC) scheme to optimise the permutation matrices for parameter interpolation by coordinating aggregated messages, and the Child Message Calibration (CMC) scheme to mitigate over-smoothing identified in PMC by calibrating the message statistics within child GNNs. Experiments across diverse domains, including node and graph property prediction, 3D object recognition, and large-scale semantic parsing, demonstrate that the proposed DuMCC effectively enables training-free knowledge transfer, yielding results on par with those of pre-trained models.
BibTeX
@inproceedings{jing2024deep,
  author = {Yongcheng Jing and Seok-Hee Hong and Dacheng Tao},
  title = {{Deep Graph Mating}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=m4NI2yIwJA},
}
Figure