Variational Graph Auto-Encoder Based Inductive Learning Method for Semi-Supervised Classification

Hanxuan Yang, Zhaoxin Yu, Qingchao Kong, Wei Liu, Wenji MaoarXiv 2024 · DOI · arXiv

Abstract

Semi-supervised inductive learning on graphs remains a significant challenge due to the scarcity of labeled data and the need for generalization to unseen graph structures. This paper introduces SLA-VGAE, a Variational Graph Auto-Encoder framework that prioritizes the reconstruction of node labels and features over graph topology. By integrating a node-masked pseudo-labeling strategy called SLAM, the model effectively leverages unlabeled data to improve classification accuracy. Experimental results on large-scale datasets like Flickr and Reddit demonstrate that SLA-VGAE outperforms existing GNN and VGAE baselines, particularly in low-label scenarios.

Introduction

Most Graph Neural Networks (GNNs) are designed for transductive learning, where the model predicts labels for unlabeled nodes within a fixed, known graph. However, real-world applications often require inductive learning—the ability to generalize to entirely new nodes or graph structures. Standard GNNs are typically "label-hungry," making them difficult to deploy when annotation is expensive. While Variational Graph Auto-Encoders (VGAEs) offer better generalization through generative regularization, they traditionally focus on reconstructing graph edges, which can lead to "proximity overfitting" and suboptimal classification performance.

Methodology

To address these limitations, the authors propose two primary innovations: the SLA-VGAE architecture and the SLAM augmentation method.

SLA-VGAE Architecture

Unlike traditional VGAEs, SLA-VGAE shifts the reconstruction objective. The encoder uses a standard GCN to produce latent representations ($Z$). The decoder then utilizes two distinct heads to reconstruct node labels ($\hat{Y}$) and node features ($\hat{X}$). By forcing the model to reconstruct labels, the latent space is optimized specifically for classification tasks rather than mere topological memorization.

Self-Label Augmentation Method (SLAM)

To mitigate label scarcity, SLAM employs a two-stage training process. After an initial warm-up phase, the model generates pseudo-labels for unlabeled nodes using node-wise masking. By randomly hiding nodes and predicting their labels based on neighbors over multiple iterations, the model learns robust structural patterns. Only predictions exceeding a specific confidence threshold ($\theta$) are retained as augmented labels.

Figure 3: Overview of the SLA-VGAE architecture and the SLAM pseudo-labeling process.

Experimental Results

The model was evaluated on the Flickr and Reddit datasets, focusing on "low-label" regimes (1%, 10%, and 100% labeling rates).

SLA-VGAE consistently demonstrated superior performance and robustness. For instance, on the Flickr dataset with only 1% of labels, SLA-VGAE achieved an accuracy of 0.475, significantly outperforming standard GCNs (0.428) and other generative models like MaskGAE (0.427). On the Reddit dataset, the model showed remarkable stability; while competing models like GAMLP saw performance drops of approximately 35% when moving from 100% to 1% labeling, SLA-VGAE's accuracy remained highly resilient.

Figure 5: Accuracy vs. Labeling Rate on the Reddit dataset, highlighting SLA-VGAE's robustness in low-label scenarios.

Discussion and Conclusion

The success of SLA-VGAE highlights three key insights:

  1. Label Reconstruction: Prioritizing label and feature reconstruction over edge reconstruction creates a latent space better suited for semi-supervised classification.
  2. Robust Masking: Node-wise masking prevents the model from memorizing labels, forcing it to learn the underlying graph logic.
  3. Generative Regularization: The KL divergence term acts as a data-agnostic regularizer, preventing weight collapse when labels are scarce.

While the method introduces some computational overhead due to the iterative pseudo-labeling process, its ability to maintain high accuracy with minimal supervision makes it a powerful tool for large-scale, real-world inductive graph learning.

Summary updated

Abstract

Graph representation learning is a fundamental research issue in various domains of applications, of which the inductive learning problem is particularly challenging as it requires models to generalize to unseen graph structures during inference. In recent years, graph neural networks (GNNs) have emerged as powerful graph models for inductive learning tasks such as node classification, whereas they typically heavily rely on the annotated nodes under a fully supervised training setting. Compared with the GNN-based methods, variational graph auto-encoders (VGAEs) are known to be more generalizable to capture the internal structural information of graphs independent of node labels and have achieved prominent performance on multiple unsupervised learning tasks. However, so far there is still a lack of work focusing on leveraging the VGAE framework for inductive learning, due to the difficulties in training the model in a supervised manner and avoiding over-fitting the proximity information of graphs. To solve these problems and improve the model performance of VGAEs for inductive graph representation learning, in this work, we propose the Self-Label Augmented VGAE model. To leverage the label information for training, our model takes node labels as one-hot encoded inputs and then performs label reconstruction in model training. To overcome the scarcity problem of node labels for semi-supervised settings, we further propose the Self-Label Augmentation Method (SLAM), which uses pseudo labels generated by our model with a node-wise masking approach to enhance the label information. Experiments on benchmark inductive learning graph datasets verify that our proposed model archives promising results on node classification with particular superiority under semi-supervised learning settings.
BibTeX
@misc{yang2024variational,
  author = {Hanxuan Yang and Zhaoxin Yu and Qingchao Kong and Wei Liu and Wenji Mao},
  title = {{Variational Graph Auto-Encoder Based Inductive Learning Method for Semi-Supervised Classification}},
  year = {2024},
  eprint = {2403.17500},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.17500},
}
Figure