Uncertainty in Graph Contrastive Learning with Bayesian Neural Networks

Alexander Möllers, Alexander Immer, Elvin Isufi, Vincent Fortuin

Abstract

Graph contrastive learning (GCL) has emerged as a powerful paradigm for semi-supervised node classification, leveraging large unlabeled datasets. However, standard GCL methods typically provide point estimates, lacking the ability to quantify uncertainty—a critical requirement for high-stakes applications. We introduce Variational Graph Contrastive Learning (VGCL), a framework that incorporates epistemic uncertainty by treating neural network weights as distributions. Furthermore, we propose the Contrastive Model Disagreement Score (CMDS), a novel uncertainty metric based on the variance in likelihood across different positive augmentations. Our results on citation networks demonstrate that VGCL improves both classification accuracy and the reliability of uncertainty estimates.

Introduction

In domains like drug discovery and social network analysis, understanding when a model is uncertain is as vital as the prediction itself. While GCL effectively learns representations by contrasting different views of data, existing models are often overconfident and fail to account for model weight uncertainty. This work addresses this gap by applying Variational Bayesian inference to the GCL objective, providing a principled way to capture epistemic uncertainty and improve downstream performance.

Methodology

Variational Graph Contrastive Learning (VGCL)

VGCL extends the InfoNCE loss by interpreting it through a generative lens and applying a Variational Bayesian approach. Instead of learning deterministic weights, the model learns the parameters (mean and variance) of a Gaussian distribution for every weight in the network. The training objective is an Evidence Lower Bound (ELBO). A key innovation is the use of Gaussian hyperpriors on the variational parameters, which prevents the model from collapsing into a deterministic state and ensures the preservation of meaningful weight variance.

Contrastive Model Disagreement Score (CMDS)

To quantify uncertainty, we propose CMDS. The core idea is that a model confident in its representation of a node should assign consistent likelihoods to that node across various stochastic augmentations (e.g., edge dropping or feature masking). High variance in these likelihoods indicates model uncertainty. Mathematically, CMDS is shown to be robust to distribution shifts, making it a reliable metric across different datasets.

Experimental Evaluation

We evaluated VGCL on three standard citation network benchmarks: Cora, CiteSeer, and PubMed. These datasets involve predicting the category of scientific papers based on their features and citation links.

Results and Discussion

Representation Quality

Visualizing the learned embeddings using t-SNE reveals that VGCL produces significantly more distinct and well-separated clusters compared to standard InfoNCE. By accounting for uncertainty during training, the model learns more robust features that are less susceptible to noise in the graph structure.

t-SNE visualization comparing InfoNCE (left) and VGCL (right) embeddings. VGCL shows clearer cluster separation.

Uncertainty Calibration and Accuracy

VGCL consistently outperforms baselines in node classification accuracy. For instance, on the Cora dataset, accuracy increased from 81.9% to 83.5%. To assess uncertainty calibration, we utilized retention curves, which track mean accuracy as the most uncertain samples are progressively removed.

Retention curves comparing different uncertainty metrics and models. The right plot shows that VGCL combined with the CMDS metric provides superior error prediction.

As shown in the retention curves, the CMDS metric serves as a much better predictor of potential errors than standard baselines like Average Standard Deviation (ASTD). The steep decline in accuracy as more samples are included (sorted by increasing uncertainty) indicates that the model's uncertainty scores are well-aligned with its actual performance.

Conclusion

This work demonstrates that bringing Bayesian uncertainty to graph contrastive learning via VGCL and the CMDS metric enhances both model performance and reliability. While currently focused on graph data and Gaussian weight distributions, future work will explore the scalability of this approach to massive graphs and its applicability to other modalities like computer vision and natural language processing.

Summary updated

Abstract

Graph contrastive learning has shown great promise when labeled data is scarce, but large unlabeled datasets are available. However, it often does not take uncertainty estimation into account. We show that a variational Bayesian neural network approach can be used to improve not only the uncertainty estimates but also the downstream performance on semi-supervised node-classification tasks. Moreover, we propose a new measure of uncertainty for contrastive learning, that is based on the disagreement in likelihood due to different positive samples.
BibTeX
@inproceedings{mollers2023uncertainty,
  author = {Alexander Möllers and Alexander Immer and Elvin Isufi and Vincent Fortuin},
  title = {{Uncertainty in Graph Contrastive Learning with Bayesian Neural Networks}},
  booktitle = {AABI},
  year = {2023},
  url = {https://openreview.net/forum?id=LH76pl-OUj},
}
Figure