When Does Self-Supervision Help Graph Convolutional Networks?

Yuning You, Tianlong Chen, Zhangyang Wang, Yang Shen

Abstract

Self-supervision as an emerging technique has been employed to train convolutional neural networks (CNNs) for more transferrable, generalizable, and robust representation learning of images. Its introduction to graph convolutional networks (GCNs) operating on graph data is however rarely explored. In this study, we report the first systematic exploration and assessment of incorporating self-supervision into GCNs. We first elaborate three mechanisms to incorporate self-supervision into GCNs, analyze the limitations of pretraining & finetuning and self-training, and proceed to focus on multi-task learning. Moreover, we propose to investigate three novel self-supervised learning tasks for GCNs with theoretical rationales and numerical comparisons. Lastly, we further integrate multi-task self-supervision into graph adversarial training. Our results show that, with properly designed task forms and incorporation mechanisms, self-supervision benefits GCNs in gaining more generalizability and robustness.

Introduction

Graph Convolutional Networks (GCNs) have revolutionized node classification, yet they often face challenges in semi-supervised settings where labeled data is scarce. While self-supervised learning (SSL) has proven highly effective in computer vision for learning from unlabeled data, its application to graph-structured data—which combines node features with complex topological structures—remains a nascent field. This study addresses how SSL can be systematically integrated into GCNs to improve generalizability and adversarial robustness.

Methodology

The researchers explored three primary mechanisms for incorporating SSL into GCN architectures: pretraining & finetuning, self-training, and multi-task learning (MTL). Unlike computer vision, where pretraining is standard, this study found that MTL is the most effective for GCNs. This is attributed to the shallow nature of typical GCNs, where information from a pretraining phase is easily lost during subsequent finetuning.

Three novel SSL pretext tasks were designed specifically for graphs:

  1. Node Feature Clustering: Groups nodes based on their feature similarity.
  2. Graph Topology Partitioning: Groups nodes based on the graph's structure to minimize inter-group edges, providing a global smoothness prior.
  3. Graph Completion: A regression task where the model must reconstruct masked node features using information from neighboring nodes.
Overview of the SS-GCN framework illustrating the integration of self-supervision via multi-task learning and the three proposed pretext tasks.

Experimental Results

The proposed methods were evaluated on standard citation benchmarks: Cora, Citeseer, and PubMed.

Generalizability

Integrating SSL tasks via multi-task learning consistently improved classification accuracy. For example, on the PubMed dataset, a standard GCN achieved 79.10% accuracy, which rose to 80.00% when augmented with Graph Partitioning (MTL-Par).

Adversarial Robustness

Self-supervision acts as a powerful shield against adversarial attacks. By forcing the model to solve a pretext task like feature reconstruction, it learns more fundamental data representations. In experiments on the Cora dataset under combined link and feature attacks, standard Adversarial Training (AdvT) yielded 39.08% accuracy. Adding the Graph Completion task (AdvT+Comp) significantly boosted this to 47.14%.

The Graph Completion task requires the model to reconstruct masked node features, enhancing the learning of robust representations.

Conclusion

This systematic study demonstrates that self-supervision, particularly when implemented through multi-task learning, significantly enhances GCNs. Tasks like Graph Partitioning and Graph Completion serve as effective data-driven regularizers, improving both the generalizability of models with few labels and their resilience to adversarial perturbations. Future research will focus on scaling these techniques to larger, industrial-scale graphs.

Summary updated

Abstract

Self-supervision as an emerging technique has been employed to train convolutional neural networks (CNNs) for more transferrable, generalizable, and robust representation learning of images. Its introduction to graph convolutional networks (GCNs) operating on graph data is however rarely explored. In this study, we report the first systematic exploration and assessment of incorporating self-supervision into GCNs. We first elaborate three mechanisms to incorporate self-supervision into GCNs, analyze the limitations of pretraining & finetuning and self-training, and proceed to focus on multi-task learning. Moreover, we propose to investigate three novel self-supervised learning tasks for GCNs with theoretical rationales and numerical comparisons. Lastly, we further integrate multi-task self-supervision into graph adversarial training. Our results show that, with properly designed task forms and incorporation mechanisms, self-supervision benefits GCNs in gaining more generalizability and robustness. Our codes are available at https://github.com/Shen-Lab/SS-GCNs.
BibTeX
@inproceedings{you2020when,
  author = {Yuning You and Tianlong Chen and Zhangyang Wang and Yang Shen},
  title = {{When Does Self-Supervision Help Graph Convolutional Networks?}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=RWYa8swkkx},
}
Figure