Self-supervised learning (SSL) has revolutionized machine learning by training models on massive unlabeled datasets using custom-made "pretext" tasks. Yet, for years, its empirical success felt like a collection of disparate engineering recipes. Why do contrastive methods (which pull augmented views of the same image together and push different images apart) and non-contrastive methods (which only look at positive pairs but use architectural tricks) both yield excellent representations?
Recent theoretical work has begun to bridge this gap, revealing that self-supervised learning is not just a set of heuristics. Instead, it is a mathematically unified framework for recovering the underlying geometry, spectral properties, and information structure of the data manifold.
The Information-Theoretic Lens: Maximizing Mutual Information
How do we formalize the idea of "learning what matters" without human labels? One powerful framework is Mutual Information (MI) maximization. If we take two different views of the same data point—such as two random crops of an image, or a sentence with certain words omitted—a good representation should capture the core semantic information shared between them while discarding independent noise.
Lingpeng Kong et al. demonstrated in A Mutual Information Maximization Perspective of Language Representation Learning that state-of-the-art language models like BERT and Skip-gram are implicitly maximizing a lower bound on the mutual information between different segments of text. By framing language pretraining as MI maximization, disparate objectives like masked language modeling can be understood under a single theoretical umbrella.
In the vision domain, non-contrastive methods like VICReg achieve this by explicitly managing the statistical properties of the embeddings. As shown by Randall Balestriero et al. in What Do We Maximize in Self-Supervised Learning? and further extended by Ravid Shwartz-Ziv and Yann LeCun in An Information Theory Perspective on Variance-Invariance-Covariance Regularization, these variance-preservation objectives maximize the entropy of the representations (preventing them from shrinking to a single point) while keeping them invariant across augmentations.
Akhlaqur Rahman Sabby et al. unified these ideas in Self-Supervised Representation Learning as Mutual Information Maximization, proving that empirical heuristics like stop-gradients, predictors, and statistical regularizers are not arbitrary tricks; they are mathematical necessities required to optimize joint mutual information bounds.
The Geometric Lens: Similarity Graphs and Spectral Embeddings
Another way to understand SSL is through graph theory and classical manifold learning. Imagine a graph where every data point (or augmented view) is a node, and an edge connects two nodes if they are augmentations of the same underlying image. This is a similarity graph.
Yang Yuan proved in Contrastive Learning is Spectral Clustering on Similarity Graph that the popular InfoNCE loss used in contrastive learning is mathematically equivalent to performing spectral clustering on this similarity graph. By minimizing the contrastive loss, the neural network learns to map nodes that are close on the graph to nearby points in the continuous embedding space.
This connects modern deep learning directly to classical dimensionality reduction. Randall Balestriero and Yann LeCun showed in Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods that both contrastive and non-contrastive methods act as solvers for global and local spectral embedding problems.
Similarly, Marina Munkhoeva et al. in Neural Harmonics: Bridging Spectral Embedding and Matrix Completion in Self-Supervised Learning demonstrated that SSL objectives solve a low-rank matrix completion problem to recover the eigenfunctions of the data manifold's Laplace operator. This explains why models can learn smooth, continuous representations from only a tiny fraction of all possible data pairs.
Furthermore, Tianyang Hu et al. established in Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding that contrastive learning implicitly performs stochastic neighbor embedding (SNE), mapping high-dimensional data manifolds into well-behaved lower-dimensional spaces.
Resolving the Collapse: The Duality of Contrastive and Non-Contrastive SSL
Historically, researchers divided SSL into two camps:
- Contrastive methods (like SimCLR) that require negative samples to prevent representations from collapsing into a single constant vector.
- Non-contrastive methods (like SimSiam or BYOL) that avoid collapse without negative pairs, relying instead on architectural asymmetries like stop-gradients and predictors.
A major breakthrough came with the realization that these two families are fundamentally dual. In On the duality between contrastive and non-contrastive self-supervised learning, the authors proved that contrastive and covariance-based losses are two sides of the same mathematical coin. The performance differences observed in practice stem from optimization nuances and hyperparameter tuning rather than distinct algorithmic mechanisms.
But how exactly do negative-free methods avoid collapse without explicit negative pairs? In How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning, the authors used a vector decomposition framework to show that asymmetric architectures implicitly perform two operations: de-centering and dimensional de-correlation.
- De-centering prevents the representations from collapsing to a single point.
- De-correlation prevents all dimensions from encoding the same feature, forcing the model to distribute information across the embedding space.
Thus, non-contrastive architectures achieve the exact same geometric regularization that negative samples provide in contrastive learning.
Why Pretext Tasks Lead to Generalization
The ultimate goal of SSL is to produce representations that perform well on unseen downstream tasks. Why does solving an artificial pretext task—like predicting a missing patch of an image or a masked word—translate to high-level semantic understanding?
The answer lies in how augmentations and pretext tasks isolate meaningful features from noise. Zixin Wen showed in Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning that raw data is often dominated by "spurious dense noise." Clever data augmentations break the correlation of this noise between positive pairs, forcing the neural network to decouple the noise and focus exclusively on the sparse, semantically meaningful features that remain consistent across views.
This mechanism is formalized by the minimal variations principle proposed by Alberto Bietti et al. in On minimal variations for unsupervised representation learning, which argues that useful representations vary slowly in regions where data is densely populated. By enforcing consistency across variations, the model learns the true coordinates of the data manifold.
Mathematically, this guarantees downstream performance. Jason D. Lee et al. proved in Predicting What You Already Know Helps: Provable Self-Supervised Learning that if two parts of an input are approximately independent given the downstream label, solving the pretext task forces the model to learn representations that make downstream tasks linearly separable.
Finally, Daniel D. Johnson et al. demonstrated in Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions that contrastive learning acts as a minimax-optimal basis creator. It guarantees the best possible foundation for predicting any downstream label that remains consistent across different augmented views of the same data.