GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference

Mahsa Ghorbani, Mojtaba Bahrami, Anees Kazi, Mahdieh Soleymani Baghshah, Hamid R. Rabiee, Nassir Navab

Introduction

In the domain of medical artificial intelligence, the integration of multi-modal data—such as functional Magnetic Resonance Imaging (fMRI) and phenotypic metadata (age, sex, genetic markers)—is essential for robust diagnostic models. Graph Convolutional Networks (GCNs) have emerged as a powerful tool in this space, treating patients as nodes in a population graph where edges represent clinical or demographic similarities. By performing graph convolutions, these models "smooth" features across related patients, significantly enhancing performance in semi-supervised learning tasks.

However, standard GCNs suffer from a critical limitation: Graph Dependency. These models require the full graph structure to be present during both training and inference. In clinical practice, a new patient may arrive at a facility where the specific metadata required to establish graph edges is unavailable, or the patient may effectively act as an "isolated node." When a GCN encounters such isolated nodes, its performance typically collapses because the "raw" features of the new patient are statistically inconsistent with the "smoothed" features the model learned during training. This mismatch makes the inference data appear as noise, rendering the graph-aware model ineffective for real-world, independent deployment.

Methodology: The GKD Framework

To address the graph dependency problem, we present Graph Knowledge Distillation (GKD). This framework utilizes a teacher-student architecture to decouple the benefits of relational graph learning from the requirements of the inference environment.

1. The Graph-Aware Teacher

The teacher model is designed to capture the relational structure of the training population. The process begins by training a standard Deep Neural Network (DNN) on labeled node features to generate initial "soft pseudo-labels" for the unlabeled portion of the dataset. These labels are then refined using a Modified Label Propagation Algorithm (LPA).

Unlike traditional LPA, which can over-rely on graph structure and "forget" the underlying node features, the GKD teacher incorporates a remembrance term. This term ensures a balance between the individual patient’s features and the relational insights provided by the graph, resulting in high-quality, graph-informed training targets.

2. The Graph-Independent Student

The student is a simpler DNN architecture that operates solely on individual patient features, with no access to the graph structure. During training, the student is tasked with mimicking the teacher’s soft pseudo-labels using a cross-entropy loss function. By learning from these "soft" probability distributions rather than "hard" binary labels, the student implicitly inherits the relational knowledge and class-boundary nuances that the teacher derived from the graph.

3. Inference

At the point of deployment, the teacher model and the population graph are discarded. The student model performs inference as a standalone classifier, requiring only the features of the individual patient being diagnosed. This allows for high-performance prediction even in settings where the broader patient-association graph cannot be constructed.

Experimental Evaluation

The GKD framework was evaluated across three distinct datasets to test its accuracy and robustness:

  • ABIDE: A dataset for Autism Spectrum Disorder (ASD) diagnosis involving fMRI and phenotypic data for 871 patients.
  • TADPOLE: A subset of the Alzheimer’s Disease Neuroimaging Initiative (ADNI) used to predict disease progression based on 15 multi-modal biomarkers (MRI, PET, CSF, etc.).
  • Synthetic Datasets: Custom data used to stress-test the model’s sensitivity to missing values in the features used for graph construction.

Results and Discussion

Performance in Graph-Missing Scenarios

GKD demonstrated a clear advantage over state-of-the-art (SOTA) graph-based methods when the graph structure was unavailable at test time. On the ABIDE dataset, GKD achieved an accuracy of 0.667, significantly outperforming EV-GCN (0.627) and the model proposed by Parisot et al. (0.60). This confirms that the student model successfully internalizes the graph's benefits without requiring its presence during inference.

Robustness and Stability

On the TADPOLE dataset, GKD reached an accuracy of 0.848, compared to 0.766 for EV-GCN. Furthermore, GKD exhibited superior stability in Macro F1 scores, suggesting that the distillation process helps the model manage class imbalances more effectively than standard graph-based approaches. The consistent gains in Area Under the Curve (AUC) across all experiments indicate that GKD learns a more reliable latent representation that is less sensitive to specific decision thresholds.

The "Smoothing" Mismatch

A key insight from this research is the identification of the "smoothing mismatch." GCNs fail on isolated nodes because they are trained to expect features that have been averaged with their neighbors. GKD bypasses this by training the student on raw features from the outset, while using the teacher's labels to guide the student toward the correct relational context.

Conclusion and Future Work

GKD provides a viable path for deploying the power of graph-based learning in real-world clinical settings where data may be incomplete. By treating soft labels as carriers of relational knowledge, we can train graph-independent models that retain the competitive edge of GCNs.

Future research will focus on automating the graph construction process, which currently relies on manual heuristics. Additionally, exploring more complex student architectures, such as 1D-CNNs for temporal medical signals, and utilizing full GCNs as teachers may further enhance the distillation of complex relational knowledge.

Summary updated

Abstract

The increased amount of multi-modal medical data has opened the opportunities to simultaneously process various modalities such as imaging and non-imaging data to gain a comprehensive insight into the disease prediction domain. Recent studies using Graph Convolutional Networks (GCNs) provide novel semi-supervised approaches for integrating heterogeneous modalities while investigating the patients’ associations for disease prediction. However, when the meta-data used for graph construction is not available at inference time (e.g., coming from a distinct population), the conventional methods exhibit poor performance. To address this issue, we propose a novel semi-supervised approach named GKD based on the knowledge distillation. We train a teacher component that employs the label-propagation algorithm besides a deep neural network to benefit from the graph and non-graph modalities only in the training phase. The teacher component embeds all the available information into the soft pseudo-labels. The soft pseudo-labels are then used to train a deep student network for disease prediction of unseen test data for which the graph modality is unavailable. We perform our experiments on two public datasets for diagnosing Autism spectrum disorder, and Alzheimer’s disease, along with a thorough analysis on synthetic multi-modal datasets. According to these experiments, GKD outperforms the previous graph-based deep learning methods in terms of accuracy, AUC, and Macro F1.
BibTeX
@inproceedings{ghorbani2021gkd,
  author = {Mahsa Ghorbani and Mojtaba Bahrami and Anees Kazi and Mahdieh Soleymani Baghshah and Hamid R. Rabiee and Nassir Navab},
  title = {{GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  doi = {10.1007/978-3-030-87240-3\_68},
}
Figure