Abstract
Recently, a variety of methods under the name of non-contrastive learning (like BYOL, SimSiam, SwAV, DINO) show that when equipped with some asymmetric architectural designs, aligning positive pairs alone is sufficient to attain good performance in self-supervised visual learning. Despite some understandings of some specific modules (like the predictor in BYOL), there is yet no unified theoretical understanding of how these seemingly different asymmetric designs can all avoid feature collapse, particularly considering methods that also work without the predictor (like DINO). In this work, we propose a unified theoretical understanding for existing variants of non-contrastive learning. Our theory named Rank Differential Mechanism (RDM) shows that all these asymmetric designs create a consistent rank difference in their dual-branch output features. This rank difference will provably lead to an improvement of effective dimensionality and alleviate either complete or dimensional feature collapse. Different from previous theories, our RDM theory is applicable to different asymmetric designs (with and without the predictor), and thus can serve as a unified understanding of existing non-contrastive learning methods. Besides, our RDM theory also provides practical guidelines for designing many new non-contrastive variants. We show that these variants indeed achieve comparable performance to existing methods on benchmark datasets, and some of them even outperform the baselines.
Introduction
In self-supervised learning (SSL), the goal is to learn useful representations from unlabeled data. Early "contrastive" methods, such as SimCLR, prevented the model from outputting a constant vector for every image (complete collapse) by explicitly pushing different images apart using negative samples. However, a newer generation of "non-contrastive" methods—including BYOL, SimSiam, and DINO—only utilize positive pairs (different views of the same image) and yet successfully avoid collapse.
While researchers have proposed various heuristics to prevent this—such as predictors, momentum encoders, and stop-gradient operations—a unified theory explaining why these diverse asymmetric designs all work has been missing. Furthermore, even when a model avoids complete collapse, it often suffers from dimensional collapse, where features inhabit only a tiny, low-dimensional subspace. This paper introduces the Rank Differential Mechanism (RDM) to provide a single theoretical framework explaining how these methods maintain high-dimensional feature diversity.
The Rank Differential Mechanism (RDM)
The RDM theory posits that asymmetric architectural designs act as spectral filters that create a "rank ladder." This mechanism consistently drives the network to increase its feature diversity and avoid dimensional collapse.
Spectral Analysis and the Rank Ladder
The authors treat asymmetric components (like the predictor head in BYOL or the sharpening operation in DINO) as filters that modify the eigenvalues of the feature covariance matrix. Successful asymmetric designs act as:
- Low-pass filters on the online branch (amplifying large eigenvalues/low-frequency components).
- High-pass filters on the target branch.
By creating a consistent difference in the "effective rank" between the two branches, the model generates a gradient that forces the online branch to "catch up" to the higher-rank target branch. This process effectively allows the model to climb a ladder toward higher dimensionality.
The Role of Stop-Gradient
The stop-gradient operation serves as a crucial anchor in this process. It prevents the target branch from "collapsing down" to match the lower rank of the online branch, ensuring that the rank differential always pulls the representations toward higher complexity rather than lower simplicity.
Empirical Validation and Filter Design
To test the RDM theory, the authors replaced learnable predictors with fixed, hand-crafted spectral filters using Singular Value Decomposition (SVD). These filters were based on functions such as $log(1+\sigma)$ or $\sigma^{-0.5}$.
Key Insights
- Asymmetry is Essential: A perfectly symmetric version of SimSiam ("SymSimSiam") was found to avoid complete collapse but still suffered from severe dimensional collapse. This confirms that asymmetry is specifically required to maintain feature diversity.
- Unified View: Whether using a learnable predictor (BYOL), Sinkhorn iterations (SwAV), or centering and sharpening (DINO), all these techniques induce a rank difference between the online and target outputs.
- Predictors as Denoise-ers: Theoretically, an optimal predictor learns to eliminate augmentation noise, making it a natural low-pass filter that focuses on stable, high-variance features.
Results
The authors validated their theory on standard benchmarks including CIFAR-10/100, ImageNet-100, and ImageNet-1k.
- Performance Gains: On CIFAR-10, SimSiam equipped with hand-crafted "target predictors" (e.g., $h(\sigma) = \sigma^{-0.5}$) achieved 78.17% accuracy, significantly outperforming the standard learnable SimSiam baseline of 74.00%.
- Scalability: On ImageNet-1k, fixed spectral filters achieved 67.73% top-1 accuracy, nearly matching the standard learnable predictor (67.97%). This suggests that the underlying principle of the filter is more important than the specific learned weights.
- Efficiency: Target-branch predictors are computationally efficient as they do not require backpropagation through the filtering operations.
Conclusion and Future Work
The Rank Differential Mechanism provides a robust, unified explanation for the success of non-contrastive learning. By viewing asymmetric designs as spectral filters that maintain a rank ladder, researchers can now design more principled SSL architectures.
While the current implementation of fixed filters requires SVD—which can be computationally expensive for extremely high dimensions—the theoretical foundation opens new doors. Future work may involve extending RDM to other domains like Natural Language Processing (NLP) and Graph Neural Networks, as well as bridging the gap between this linear spectral theory and the complex non-linear dynamics of deep networks.
Summary updated