TL;DR
This paper introduces Variational Graph Contrastive Learning (VGCL), framing graph contrastive learning through full-network Bayesian neural networks (BNNs) with regularized hyperpriors. Alongside this framework, the authors propose the Contrastive Model Disagreement Score (CMDS), a principled uncertainty metric based on likelihood disagreement across different data augmentations and parameter samples. Together, these methods improve both downstream semi-supervised classification accuracy and uncertainty calibration over existing baselines.
What Problem Does This Paper Address?
Contrastive learning on graphs—often using the InfoNCE objective—excels at representation learning when labeled data is scarce. However, most existing methods are deterministic and struggle with uncertainty estimation, which is vital in high-stakes graph applications like drug discovery.
Prior attempts to quantify uncertainty in contrastive learning have mostly measured the variance of node representations directly in embedding space (e.g., embedding feature standard deviation). The authors argue that this misses the true generative nature of contrastive models. Furthermore, existing Bayesian approaches often restrict inference to only the final layer or focus solely on learning augmentation parameters, failing to capture epistemic uncertainty across the full network.
Methods
The paper builds on the generative view of InfoNCE (which views the contrastive loss as an Evidence Lower Bound, or ELBO):
-
Variational Graph Contrastive Learning (VGCL):
Extends the contrastive ELBO to place variational distributions over all model weights $w = (\phi, \rho)$—both the graph convolutional encoder and the projection head. -
Employs mean-field Gaussian variational inference with reparameterization: $\sigma_v = \log(1 + \exp(p))$.
- Places Gaussian hyperpriors over the variational parameters ($\mu_v \sim \mathcal{N}(0, \sigma_0^2)$, $p \sim \mathcal{N}(\mu_p, \sigma_p^2)$) to encourage larger weight variance, accounting for the fact that augmentations are only imperfect approximations of true data-generating distributions.
- Contrastive Model Disagreement Score (CMDS):
Inspired by out-of-distribution detection in Bayesian VAEs, CMDS quantifies the consistency of the model's likelihood across $M$ sampled augmentations $x'_j$ and sampled network weights $(\phi_j, \rho_j)$: $$D_{\text{CMDS}}(x) = \frac{1}{M \sum_{j=1}^M l_j(x)^2}, \quad l_j(x) = \frac{P(x \mid x'_j, \phi_j, \rho_j)}{\sum_{i=1}^M P(x \mid x'_i, \phi_i, \rho_i)}$$ - If the model reliably explains a data point across different views, the normalized likelihoods $l_j(x)$ are uniform, yielding a high disagreement score (indicating high certainty). If explanations conflict, the score drops toward its minimum of $1/M$.
- The authors prove that the ratio in CMDS cancels out dependencies on the intractable true data distribution $P_{\text{true}}(x)$.
Datasets
The authors evaluate their approach on three standard Planetoid citation network benchmarks for semi-supervised node classification:
- Cora: 2,708 scientific publications (nodes) with citation links (edges).
- CiteSeer: 3,327 nodes.
- Pubmed: 19,717 nodes.
No new datasets were introduced. A standard transductive 10% train / 10% validation / 80% test split was used with a two-layer GCN encoder.
Key Insights and Ideas
- Contrastive Learning is Generative: The InfoNCE loss can be derived directly as an ELBO without needing an explicit decoder or reconstruction loss. Integrating a BNN naturally equips contrastive representations with epistemic uncertainty.
- Hyperpriors Promote Meaningful Latent Geometry: Enforcing larger weight variance via hyperpriors regularizes representations, producing visibly tighter and cleaner class separation in embedding space (validated via t-SNE).
- Likelihood Disagreement Beats Embedding Variance: Measures operating in the embedding space (like ASTD or PSFV) miss uncertainty introduced by the projection head. Evaluating disagreement in likelihood space ($P(x \mid x')$) correlates much better with actual downstream prediction errors.
Key Results
- Downstream Accuracy: VGCL outperformed standard InfoNCE, variational autoencoders (GAE, VGAE), and mutual information methods (DGI) across all benchmarks:
- Cora: 83.5% ± 0.2 (vs. 81.9% InfoNCE, 82.6% DGI)
- CiteSeer: 72.2% ± 0.1 (vs. 70.8% InfoNCE, 68.8% DGI)
- Pubmed: 86.3% ± 0.1 (vs. 85.0% InfoNCE, 86.0% DGI)
- Uncertainty Calibration: Tested via error retention curves (measuring accuracy as progressively more uncertain samples are added):
- CMDS systematically yielded the best sample sorting compared to embedding feature variance (PSFV, ASTD) and raw likelihood/WAIC baselines.
- Adding full Bayesian weight uncertainty (VGCL) notably boosted retention curves over deterministic InfoNCE.
Limitations and Future Work
- Evaluation Scope: Experiments were confined to standard transductive citation benchmarks using a simple two-layer GCN architecture.
- Computational Overhead: Evaluating the BNN and computing CMDS requires sampling multiple augmentations and multiple sets of network weights, increasing inference time relative to standard deterministic models.
- Future Directions:
- Exploring non-Gaussian priors and more expressive variational distributions.
- Testing whether the combination of VGCL and CMDS generalizes beyond graphs to other domains, such as vision or multimodal self-supervised learning.
Summary updated