Understanding InfoNCE: Transition Probability Matrix Induced Feature Clustering

Ge Cheng, Shuo Wang, Yun Zhang

TL;DR

This paper demystifies the popular InfoNCE loss by showing it drives feature clustering toward a specific target defined by data augmentation dynamics. The authors introduce SC-InfoNCE, a modified loss function with tunable parameters that allow researchers to "dial in" the optimal level of feature similarity, leading to state-of-the-art performance across vision, text, and graph benchmarks.

What Problem Does This Paper Address?

Contrastive Learning (CL) has become the gold standard for unsupervised representation learning—think of models like SimCLR or MoCo. At its heart is the InfoNCE loss, which works by pulling "positive" pairs (different views of the same image) together and pushing "negative" pairs (views of different images) apart.

While InfoNCE works incredibly well in practice, our theoretical understanding of why it organizes the representation space the way it does is still somewhat fuzzy. Previous theories linked it to Mutual Information or spectral clustering, but they didn't fully explain how the specific choice of data augmentations (like random cropping or word masking) dictates the final "shape" of the learned features. The authors set out to bridge this gap between empirical success and theoretical intuition.

Methods

The researchers introduce a new framework based on two core concepts:

  1. Explicit Feature Space: Instead of treating features as abstract points, they define a space of all possible "views" reachable through augmentations.
  2. Transition Probability Matrix (TPM): They model augmentations as a matrix $A$, where each entry $A_{ij}$ represents the probability that a source sample $i$ will be transformed into a specific view $j$.

The "Aha!" Moment: The authors prove that InfoNCE isn't just blindly pulling and pushing; it is mathematically driving the co-occurrence probability of features toward a constant convergence target determined by this TPM.

To capitalize on this, they propose Scaled Convergence InfoNCE (SC-InfoNCE). This new loss function introduces two hyperparameters, $\delta$ (delta) and $\gamma$ (gamma), which act as "tuning knobs." These allow practitioners to scale and bias the convergence target, giving them explicit control over how tightly features should cluster or how far apart different classes should be pushed.

Datasets

The authors validated their theory and method across a wide range of modalities:

  • Vision: CIFAR-10, CIFAR-100, STL-10, and ImageNet-100.
  • Graphs: COLLAB, DD, NCI1, and PROTEINS.
  • Text: STS-B and SICK-R (evaluated via the SimCSE framework).
  • Synthetic: They also created a synthetic graph dataset with a known, ground-truth TPM to verify that their mathematical predictions about convergence actually hold true in a controlled environment.

Key Insights and Ideas

  • InfoNCE as a Target-Seeker: The paper shifts the perspective of InfoNCE from a simple "pull-push" mechanism to a process that seeks a statistical equilibrium defined by the augmentation strategy.
  • The Goldilocks Zone of Scaling: The authors found that the scale of the convergence target is critical. If the target is too small, the model becomes hypersensitive to noise; if it's too large, the representation can "collapse," where all features look the same. SC-InfoNCE allows you to find the "just right" middle ground.
  • Covariance-Based Clustering: In a simplified version of the loss (SCL), the gradient direction is actually determined by the covariance of the transition probability vectors. This provides a mechanistic explanation for why contrastive learning is so robust across different types of data.

Key Results

SC-InfoNCE consistently outperformed standard InfoNCE and several recent variants (like DCL and DHEL) across all domains:

  • Vision: On CIFAR-10, SC-InfoNCE achieved 91.49% accuracy, surpassing the standard InfoNCE baseline of 90.53%. On ImageNet-100, it reached 75.62%.
  • Graphs & Text: The method showed similar "consistent gains," often matching or exceeding strong baselines specifically designed for those domains.
  • Better Feature Organization: t-SNE visualizations (plots that project high-dimensional data into 2D) showed that SC-InfoNCE produces much tighter, more distinct clusters with less overlap between classes compared to standard methods.
  • Spectral Health: Eigenvalue analysis revealed that SC-InfoNCE utilizes the available representation space more effectively, avoiding the "dimensional collapse" that plagues simpler contrastive losses.

Limitations and Future Work

While SC-InfoNCE offers more control, the authors acknowledge a few hurdles:

  • Global vs. Local Scaling: Currently, the scaling parameters ($\delta, \gamma$) are applied globally. The authors suggest that adaptive, data-driven scaling—where the target changes based on the specific sample—could be a powerful next step.
  • Estimation Noise: The method relies on estimating transition probabilities. In complex, real-world datasets, these estimates can be noisy or asymmetric, which might prevent the model from reaching the perfect theoretical target.
  • Symmetry: Future research may look into "symmetrized" probability formulations to make the convergence even more stable and reliable.

Summary updated

Abstract

Contrastive learning has emerged as a cornerstone of unsupervised representation learning across vision, language, and graph domains, with InfoNCE as its dominant objective. Despite its empirical success, the theoretical underpinnings of InfoNCE remain limited. In this work, we introduce an explicit feature space to model augmented views of samples and a transition probability matrix to capture data augmentation dynamics. We demonstrate that InfoNCE optimizes the probability of two views sharing the same source toward a constant target defined by this matrix, naturally inducing feature clustering in the representation space. Leveraging this insight, we propose Scaled Convergence InfoNCE (SC-InfoNCE), a novel loss function that introduces a tunable convergence target to flexibly control feature similarity alignment. By scaling the target matrix, SC-InfoNCE enables flexible control over feature similarity alignment, allowing the training objective to better match the statistical properties of downstream data. Experiments on benchmark datasets, including image, graph, and text tasks, show that SC-InfoNCE consistently achieves strong and reliable performance across diverse domains.
BibTeX
@misc{cheng2025understanding,
  author = {Ge Cheng and Shuo Wang and Yun Zhang},
  title = {{Understanding InfoNCE: Transition Probability Matrix Induced Feature Clustering}},
  year = {2025},
  eprint = {2511.12180},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.12180},
}
Figure