Abstract
Graph contrastive learning (GCL) has emerged as a powerful paradigm for semi-supervised node classification, leveraging large unlabeled datasets. However, standard GCL methods typically provide point estimates, lacking the ability to quantify uncertainty—a critical requirement for high-stakes applications. We introduce Variational Graph Contrastive Learning (VGCL), a framework that incorporates epistemic uncertainty by treating neural network weights as distributions. Furthermore, we propose the Contrastive Model Disagreement Score (CMDS), a novel uncertainty metric based on the variance in likelihood across different positive augmentations. Our results on citation networks demonstrate that VGCL improves both classification accuracy and the reliability of uncertainty estimates.
Introduction
In domains like drug discovery and social network analysis, understanding when a model is uncertain is as vital as the prediction itself. While GCL effectively learns representations by contrasting different views of data, existing models are often overconfident and fail to account for model weight uncertainty. This work addresses this gap by applying Variational Bayesian inference to the GCL objective, providing a principled way to capture epistemic uncertainty and improve downstream performance.
Methodology
Variational Graph Contrastive Learning (VGCL)
VGCL extends the InfoNCE loss by interpreting it through a generative lens and applying a Variational Bayesian approach. Instead of learning deterministic weights, the model learns the parameters (mean and variance) of a Gaussian distribution for every weight in the network. The training objective is an Evidence Lower Bound (ELBO). A key innovation is the use of Gaussian hyperpriors on the variational parameters, which prevents the model from collapsing into a deterministic state and ensures the preservation of meaningful weight variance.
Contrastive Model Disagreement Score (CMDS)
To quantify uncertainty, we propose CMDS. The core idea is that a model confident in its representation of a node should assign consistent likelihoods to that node across various stochastic augmentations (e.g., edge dropping or feature masking). High variance in these likelihoods indicates model uncertainty. Mathematically, CMDS is shown to be robust to distribution shifts, making it a reliable metric across different datasets.
Experimental Evaluation
We evaluated VGCL on three standard citation network benchmarks: Cora, CiteSeer, and PubMed. These datasets involve predicting the category of scientific papers based on their features and citation links.
Results and Discussion
Representation Quality
Visualizing the learned embeddings using t-SNE reveals that VGCL produces significantly more distinct and well-separated clusters compared to standard InfoNCE. By accounting for uncertainty during training, the model learns more robust features that are less susceptible to noise in the graph structure.
Uncertainty Calibration and Accuracy
VGCL consistently outperforms baselines in node classification accuracy. For instance, on the Cora dataset, accuracy increased from 81.9% to 83.5%. To assess uncertainty calibration, we utilized retention curves, which track mean accuracy as the most uncertain samples are progressively removed.
As shown in the retention curves, the CMDS metric serves as a much better predictor of potential errors than standard baselines like Average Standard Deviation (ASTD). The steep decline in accuracy as more samples are included (sorted by increasing uncertainty) indicates that the model's uncertainty scores are well-aligned with its actual performance.
Conclusion
This work demonstrates that bringing Bayesian uncertainty to graph contrastive learning via VGCL and the CMDS metric enhances both model performance and reliability. While currently focused on graph data and Gaussian weight distributions, future work will explore the scalability of this approach to massive graphs and its applicability to other modalities like computer vision and natural language processing.
Summary updated