Abstract: Self-supervised learning (SSL) has demonstrated remarkable success in learning data representations for downstream tasks. However, theoretical understanding of this process, particularly for nonlinear models, has remained limited. Most existing works treat nonlinear networks as "black boxes" or focus on linear simplifications. This paper provides the first accurate analysis of the learning results of a nonlinear SSL model. By considering a toy data distribution with label-related and hidden features, we prove that a 1-layer nonlinear SSL model trained via gradient descent converges to a local minimum that captures both feature types. In contrast, supervised learning (SL) models primarily capture label-related features. Our analysis utilizes the Inverse Function Theorem to describe the features learned at the local minimum, and we validate these findings through simulation experiments.
Introduction
Self-supervised learning has revolutionized fields like computer vision and natural language processing by enabling models to learn useful representations without explicit labels. Despite this empirical success, the theoretical underpinnings of why SSL learns richer representations than supervised learning are still being explored. A key question is "representation learnability": the ability of a model to capture latent data structures that might be irrelevant to a current task's labels but crucial for future downstream tasks.
Methods
The researchers analyzed a 1-layer nonlinear SSL model, specifically a variant of the SimSiam architecture. This model uses a nonlinear activation function (such as Sigmoid or Tanh) and is trained to maximize the similarity between different augmented views of the same data point.
To analyze this non-convex optimization problem, the authors introduced a representative "toy" data distribution. This distribution contains two orthogonal unit-norm features: $e_1$ (label-related) and $e_2$ (hidden). While $e_1$ distinguishes the classes in a supervised setting, $e_2$ represents information that is present in the data but not necessary for classification.
Theoretical Analysis
A major challenge in analyzing nonlinear models is the lack of closed-form solutions. The authors bypassed this by using the exact version of the Inverse Function Theorem. They first analyzed a simplified, noise-free objective function and then used the theorem as a bridge to prove that the local minima of the actual, noisy objective function still effectively capture the data features. They demonstrated that standard Gradient Descent (GD) converges to these "good" local minima at a geometric rate when initialized within a specific region.
Key Insights: SSL vs. Supervised Learning
The study highlights a fundamental difference in how SSL and SL interact with data features:
- SSL as a Feature Magnet: The SSL objective forces the model's weight matrix to align with all prominent features in the data. Because it seeks to maximize similarity between augmented views, any consistent structure (like $e_2$) becomes a target for learning.
- Supervised Learning's "Tunnel Vision": Supervised learning only needs to satisfy a margin constraint to separate classes. Once the model has captured enough of the label-related feature $e_1$ to achieve low classification error, it has no mathematical incentive to learn the hidden feature $e_2$.
Key Results and Simulations
The theoretical findings are summarized in two main theorems:
- Theorem 1 (SSL Feature Capture): The SSL model captures both $e_1$ and $e_2$ almost perfectly. The projection of these features onto the learned weight space is shown to be nearly 1 ($|\Pi e| \geq 1 - O(\tau^3 d^{-1/2})$).
- Theorem 2 (SL Feature Neglect): In the SL model, the weights associated with the hidden feature $e_2$ are shown to be vanishingly small as the data dimensionality $d$ increases, proving that SL effectively ignores the hidden feature.
Simulation experiments with 20 random seeds confirmed these theorems. As shown in the learning curves, the SSL model's weights for both $e_1$ and $e_2$ components grow and stabilize at non-zero values. In contrast, the authors' SL experiments showed that hidden feature components remained near zero throughout training.
Conclusion
This work provides a rigorous mathematical foundation for the superior transferability of SSL representations. By proving that nonlinear SSL models naturally capture hidden features that supervised models ignore, the authors offer a clear explanation for why SSL is so effective for diverse downstream tasks. Future work will aim to extend this analysis to deeper, multi-layer architectures and more complex, real-world data distributions.
Summary updated