GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference

Mahsa Ghorbani, Mojtaba Bahrami, Anees Kazi, Mahdieh Soleymani Baghshah, Hamid R. Rabiee, Nassir Navab

Graph Convolutional Networks (GCNs) have become a powerhouse for analyzing multi-modal medical data. By treating patients as nodes and their phenotypic similarities (like age or gender) as edges, GCNs can "smooth" information across a population to improve disease prediction. But there is a catch: what happens when you have a new patient at inference time, but you're missing the metadata needed to slot them into your graph?

In a new paper, researchers propose GKD (Graph Knowledge Distillation), a framework that leverages the power of population graphs during training but remains completely independent of them during inference.

TL;DR

GKD uses a "Teacher" model to capture complex patient relationships from a population graph and distills that knowledge into a "Student" model. This allows the student to make high-accuracy predictions on new patients using only their primary features (like imaging), even when the metadata required for graph construction is missing.

What Problem Does This Paper Address?

Modern medical diagnosis often relies on multiple modalities—think fMRI scans (imaging) combined with genetic markers and clinical history (non-imaging). GCNs are excellent at integrating these because they can model the "context" of a patient relative to others.

However, GCNs have a major deployment hurdle: graph dependency. To classify a new patient, a standard GCN needs to know how that patient relates to the rest of the training population. If the metadata used to build those relationships (e.g., specific biomarkers or demographic info) is missing or comes from a different population, the patient becomes an "isolated node." Because GCNs are designed to work on smoothed features from neighbors, their performance drops significantly when those neighbors disappear.

Methods

The authors solve this using Knowledge Distillation, a technique where a simple "Student" model learns to mimic a complex "Teacher" model.

Fig. 1: Overview of the proposed method. The training set comprises of labeled and unlabeled nodes. Unlabeled nodes are in dark grey and other colors depicts the node labels. The upper part: teacher network. The lower part: student net- work. The student is trained by cross-entropy loss between teacher and student outputs to inject the teachers knowledge into the student’s parameters.
  1. The Teacher Component: The teacher isn't just one network; it's a pipeline. First, a standard Deep Neural Network (DNN) makes initial predictions based on node features. Then, these predictions are refined using a modified Label Propagation Algorithm (LPA) on the population graph.
  2. The Twist: The authors add a "remembrance term" to the LPA. This ensures the graph structure doesn't completely overwrite the individual patient's unique features, balancing local data with global context.
  3. The Student Component: The student is a standard DNN that never sees the graph. It is trained to minimize the cross-entropy loss between its own predictions and the "soft" pseudo-labels generated by the teacher.
  4. Inference: At test time, only the Student is used. It takes the raw, high-variance input features and maps them to the refined output space it learned from the teacher, effectively "hallucinating" the benefits of the graph without needing it.

Datasets

The researchers evaluated GKD on three fronts:

  • ABIDE: A dataset for Autism Spectrum Disorder (ASD) diagnosis using fMRI features for nodes and phenotypic data for the graph.
  • TADPOLE: A subset of the ADNI initiative for predicting Alzheimer’s progression, using 15 biomarkers (MRI, PET, CSF, etc.).
  • Synthetic Datasets: Created to rigorously test how the model handles varying rates of missing graph data during the training phase.

Key Insights and Ideas

  • Graph as a Regularizer: Instead of viewing the graph as a required input, GKD treats it as "privileged information" available only during training. This shifts the graph's role from a structural necessity to a powerful training supervisor.
  • Overcoming the "Isolated Node" Trap: Conventional GCNs fail on isolated nodes because they expect smoothed, low-variance features. By training the student to map noisy, raw features to refined labels, GKD creates a model that is inherently more robust to the lack of social/population context.
  • Soft Labels are Key: By using the teacher's probability distributions (soft labels) rather than just hard "sick/healthy" labels, the student captures the nuances and similarities between classes that the graph helped uncover.

Key Results

GKD consistently outperformed state-of-the-art graph-based methods (like EV-GCN and Parisot et al.) in scenarios where the graph was unavailable at inference time.

  • ABIDE (Autism): GKD achieved an AUC of 0.695, significantly higher than the ~0.66 range seen in competing GCN methods when the graph was removed.
  • TADPOLE (Alzheimer's): The results were even more striking. GKD reached an accuracy of 0.848, while other graph-based methods dropped to around 0.76 when forced to operate without their graphs.
  • Robustness: Synthetic experiments showed that while all models suffer as graph data becomes sparser during training, GKD maintains a higher performance floor and more stability than its peers.

Limitations and Future Work

While GKD removes the need for a graph at inference, it still requires a high-quality graph during the training/distillation phase. If the initial population graph is poorly constructed or misleading, the "knowledge" distilled into the student will be flawed.

The authors suggest that future work could explore more complex student architectures or apply this distillation framework to multi-class problems and other medical domains where data modalities are frequently missing.

For more details, you can find the full paper: "GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference" (2021).

Summary updated

Abstract

The increased amount of multi-modal medical data has opened the opportunities to simultaneously process various modalities such as imaging and non-imaging data to gain a comprehensive insight into the disease prediction domain. Recent studies using Graph Convolutional Networks (GCNs) provide novel semi-supervised approaches for integrating heterogeneous modalities while investigating the patients' associations for disease prediction. However, when the meta-data used for graph construction is not available at inference time (e.g., coming from a distinct population), the conventional methods exhibit poor performance. To address this issue, we propose a novel semi-supervised approach named GKD based on knowledge distillation. We train a teacher component that employs the label-propagation algorithm besides a deep neural network to benefit from the graph and non-graph modalities only in the training phase. The teacher component embeds all the available information into the soft pseudo-labels. The soft pseudo-labels are then used to train a deep student network for disease prediction of unseen test data for which the graph modality is unavailable. We perform our experiments on two public datasets for diagnosing Autism spectrum disorder, and Alzheimer's disease, along with a thorough analysis on synthetic multi-modal datasets. According to these experiments, GKD outperforms the previous graph-based deep learning methods in terms of accuracy, AUC, and Macro F1.
BibTeX
@inproceedings{ghorbani2021gkd,
  author = {Mahsa Ghorbani and Mojtaba Bahrami and Anees Kazi and Mahdieh Soleymani Baghshah and Hamid R. Rabiee and Nassir Navab},
  title = {{GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  doi = {10.1007/978-3-030-87240-3\_68},
}
Figure