Entropy Neural Estimation for Graph Contrastive Learning

Yixuan Ma, Xiaolin Zhang, Peng Zhang, Kun Zhan

Introduction

Unsupervised Graph Contrastive Learning (GCL) has emerged as a powerful paradigm for learning node representations without manual labeling. Traditionally, GCL frameworks operate by generating two augmented "views" of a graph—through methods like edge dropping or feature shuffling—and training a neural network to maximize the agreement between representations of the same node across these views.

Despite its success, GCL faces two significant challenges. First, performance is often highly sensitive to the choice of data augmentation, requiring extensive domain expertise to tune. Second, standard contrastive losses frequently suffer from the "class collision" problem, where nodes belonging to the same semantic category are treated as negative pairs, forcing the model to learn suboptimal features by pushing similar entities apart. This paper introduces M-ILBO (Maximizing Information-entropy Lower BOund), a framework designed to bypass complex augmentations and mitigate class collision through an information-theoretic approach to entropy estimation.

Methodology

The M-ILBO framework is built upon the principle that the entropy of a dataset can be estimated by maximizing the mutual information between randomly sampled subsets. The approach consists of three primary pillars:

1. Subset Sampling as Information Ensemble

Rather than relying on manually designed augmentations, M-ILBO utilizes a "bagging" inspired approach. The model samples two random subsets of nodes and edges from the original graph. By maximizing the mutual information between these subsets over multiple iterations, the model effectively estimates the total entropy of the dataset. This theoretical grounding justifies the use of simple subsetting as a robust alternative to complex, task-specific augmentations.

2. Similarity-Based Pair Selection

To address the class collision problem, the authors move beyond the "diagonal-only" approach of traditional contrastive learning. Instead of only treating node $i$ in view 1 and node $i$ in view 2 as a positive pair, M-ILBO analyzes the entire similarity matrix. It identifies "latent" positives—nodes that exhibit high similarity across views—and treats them as additional positive pairs. Conversely, it identifies nodes with very low similarity to serve as hard negatives. This strategy enriches the supervision signal and allows the model to capture category-level information in an entirely unsupervised manner.

3. Dual-Objective Loss Function

The model is optimized using a combination of two loss terms:

  • Contrastive Loss ($L_{cl}$): Utilizes Jensen-Shannon divergence to maximize the mutual information between the sampled views.
  • Cross-View Consistency ($L_{cvc}$): Employs an $\ell_2$ norm to ensure that the representations for the same nodes remain consistent regardless of the specific subset in which they were sampled.

Experimental Results

The authors evaluated M-ILBO on seven benchmark datasets, including citation networks (Cora, Citeseer, Pubmed) and co-occurrence/co-purchase networks (Amazon, Co-author).

Performance Highlights

M-ILBO achieved state-of-the-art or highly competitive results across all benchmarks, frequently outperforming both unsupervised and supervised baselines:

  • Cora: The model achieved 85.7% accuracy, surpassing the previous unsupervised leader (CCA-SSG at 84.2%) and even the supervised GCN (81.5%).
  • Citeseer: Accuracy reached 74.2%, a notable improvement over DGI (71.8%).
  • Ablation Studies: Results indicated that the similarity-based pair selection strategy provided a ~1.6% performance boost on the Cora dataset compared to standard random shuffling, confirming that the method for selecting negative pairs is critical for representation quality.

Discussion and Future Work

The core innovation of M-ILBO lies in its link between contrastive learning and information theory, framing subset sampling as a form of "Information Ensemble" similar to Bagging or Dropout. By looking beyond simple node-to-node identity, the model successfully navigates the class collision problem that plagues many GCL methods.

However, this improved accuracy comes with increased computational complexity. The process of calculating and sorting the similarity matrix to identify supervision pairs is more memory-intensive and time-consuming than simple random sampling. Future research will focus on optimizing the efficiency of this selection process to make M-ILBO more scalable for extremely large-scale graph datasets.

Summary updated

BibTeX
@inproceedings{ma2023entropy,
  author = {Yixuan Ma and Xiaolin Zhang and Peng Zhang and Kun Zhan},
  title = {{Entropy Neural Estimation for Graph Contrastive Learning}},
  booktitle = {ACM Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=q6UCzY4VBh},
}
Figure