Contrastive latent representation learning with denoising score matching

Christopher Marouani, Grigory Bartosh, Christian A. NaessetharXiv 2026 · arXiv

TL;DR

ScoreCLR is a novel contrastive learning framework that replaces the computationally expensive "negative sampling" step with a learned repulsion force estimated via denoising score matching. By moving from a quadratic $O(B^2)$ to a linear $O(B)$ complexity relative to batch size, it achieves competitive accuracy while significantly reducing memory overhead and preventing the "dimensional collapse" common in traditional methods.

What Problem Does This Paper Address?

In self-supervised learning, contrastive methods (like SimCLR) are the gold standard. They work on a simple principle: pull different "views" of the same image together (attraction) and push views of different images apart (repulsion).

However, the "repulsion" part is a major bottleneck. To prevent all images from collapsing into a single point, models compare each sample against a batch of "negatives." This approach has three big flaws:

  1. Computational Cost: Comparing every sample to every other sample in a batch scales quadratically ($O(B^2)$). Large batches are essential for performance but require massive GPU memory.
  2. High Variance: The estimate of the repulsive force depends heavily on which specific negative samples happen to be in your current batch.
  3. Dimensional Collapse: These models often end up using only a tiny fraction of the available latent space dimensions, effectively "wasting" the model's capacity.

Methods

The authors introduce ScoreCLR, which reframes representation learning as maximizing the Mutual Information (MI) between two latent views.

The core mathematical "aha!" moment is realizing that the gradient of the latent distribution's entropy—which provides the repulsive force—is equivalent to the score of that distribution ($\nabla_z \log p(z)$). Instead of calculating this score by looking at other samples in a batch, ScoreCLR trains a lightweight auxiliary "score network" to estimate it.

Key technical components include:

  • Denoising Score Matching (DSM): The score network is trained to predict the noise added to latent representations across multiple scales.
  • ScoreReg: A new regularization term derived from the score that encourages the latent space to maintain high entropy (i.e., stay spread out).
  • Gaussian Prior: An $L^2$ penalty that keeps the representations from drifting to infinity, pushing the latent distribution toward a standard normal distribution.
Figure 1: Repulsion force acting on a latent code z (negative gradient of the loss). SimCLR (dashed) estimates it against the B codes of the batch, with a cost O(B2) per batch, whereas ScoreCLR (solid) uses the score −∇z log pZ(z) of the latent distribution, learned by a small network, with a cost O(B).

Datasets

The authors validated ScoreCLR using standard computer vision benchmarks:

  • Pre-training: CIFAR-10 and ImageNet-100.
  • Downstream Transfer: STL-10 and the Describable Textures Dataset (DTD) to test how well the learned features generalize to new types of data.

Key Insights and Ideas

  • Amortized Repulsion: By "baking" the knowledge of the data distribution into a score network, the model no longer needs to see negative samples during the encoder update. The repulsion is amortized.
  • Linear Scaling is the Future: At a batch size of 32,768, ScoreCLR’s objective is 10x faster and uses 53x less memory than the standard InfoNCE loss used in SimCLR.
  • Fighting Dimensional Collapse: Traditional contrastive methods often suffer from "dimensional collapse," where the model ignores many available dimensions. ScoreCLR produces representations with a much higher effective rank, meaning it utilizes the latent space much more efficiently.

Key Results

ScoreCLR achieves performance nearly identical to SimCLR but with a much healthier latent space geometry:

  • Accuracy: On CIFAR-10, ScoreCLR reached 88.87% linear probe accuracy, trailing SimCLR (89.26%) by only a fraction of a percent.
  • Latent Rank: Using the "RankMe" metric, ScoreCLR’s projector output achieved a rank of 229/256 on CIFAR-10, compared to just 96/256 for SimCLR. This indicates that ScoreCLR's representations are far more diverse and less collapsed.
  • Uniformity: ScoreCLR shows better "uniformity" on the hypersphere for ImageNet-100, meaning it spreads features out more evenly than its competitors.
Table 1: Linear-probe and weighted k-NN (k = 20) accuracy (%) on frozen features, ResNet- 18. Top: shared optimizer and projector (AdamW, gradient clip 1.0, 2048/256 projector without BatchNorm); mean ± std over three seeds (top-5 averaged); BYOL and Barlow Twins use a projector and predictor without BatchNorm (Appendix C). Bottom: SimCLR trained with solo-learn’s LARS recipe, with the shared 2048/256 projector and with the published 4096/512 projector. With the latter, our offline linear probe gives 77.02 on ImageNet-100, against 77.64 for the online probe of the published checkpoint.

Limitations and Future Work

While promising, ScoreCLR isn't a "free lunch" yet:

  • The Accuracy Gap: There is still a small performance gap (0.4 to 1.2 points) compared to the best-tuned SimCLR baselines.
  • Hyperparameter Sensitivity: The method introduces new parameters, such as the number of score-network updates ($S$) and noise queries ($K$), which may need careful tuning for different architectures.
  • Generative Potential: The authors suggest that since they are learning the score of the latent space, the score network could potentially be used as a generative model to sample new representations directly. Testing this on larger scales like ImageNet-1K is a logical next step.

Summary updated

Abstract

Contrastive representation learning methods operate by pulling different views of the same data sample together while pushing them away from views of different samples. A fundamental challenge is the estimation of the repulsive force, which typically requires large batch sizes, can exhibit high variance, and is susceptible to the curse of dimensionality. We introduce ScoreCLR, a contrastive representation learning method that offers a novel way to estimate this repulsion term. We consider the framework of representation learning by maximizing the mutual information between two latent views of a sample. By deriving the gradient of the Shannon entropy with respect to the encoder parameters, we obtain an equivalent objective that involves the score of the latent representation. We then use denoising score matching techniques to estimate this score with a lightweight network that is trained jointly with the encoder. We further propose regularization techniques using different noise levels and a Gaussian prior on the latent representation. ScoreCLR replaces the batch of explicit negative samples in contrastive methods with a learned repulsion whose cost is linear in the batch size rather than quadratic. On CIFAR-10 and ImageNet-100 it does so with minimal loss of linear-probe accuracy compared to SimCLR under a shared recipe, while producing projector outputs of higher effective rank, closer to the Gaussian prior.
BibTeX
@misc{marouani2026contrastive,
  author = {Christopher Marouani and Grigory Bartosh and Christian A. Naesseth},
  title = {{Contrastive latent representation learning with denoising score matching}},
  year = {2026},
  eprint = {2611.18469},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18469},
}
Figure