Abstract
Self-supervised learning (SSL) has empirically shown its data representation learnability in many downstream tasks. There are only a few theoretical works on data representation learnability, and many of those focus on final data representation, treating the nonlinear neural network as a "black box". However, the accurate learning results of neural networks are crucial for describing the data distribution features learned by SSL models. Our paper is the first to analyze the learning results of the nonlinear SSL model accurately. We consider a toy data distribution that contains two features: the label-related feature and the hidden feature. Unlike previous linear setting work that depends on closed-form solutions, we use the gradient descent algorithm to train a 1-layer nonlinear SSL model with a certain initialization region and prove that the model converges to a local minimum. Furthermore, different from the complex iterative analysis, we propose a new analysis process which uses the exact version of Inverse Function Theorem to accurately describe the features learned by the local minimum. With this local minimum, we prove that the nonlinear SSL model can capture the label-related feature and hidden feature at the same time. In contrast, the nonlinear supervised learning (SL) model can only learn the label-related feature. We also present the learning processes and results of the nonlinear SSL and SL model via simulation experiments.
Introduction
Self-supervised learning (SSL) has achieved remarkable success in learning high-quality data representations without human-annotated labels. Despite this empirical progress, theoretical understanding of why SSL works—particularly in nonlinear settings—remains limited. Most existing theories either rely on simplified linear models or treat the neural network as a "black box," focusing only on the final output rather than the optimization process.
This paper addresses this gap by providing a rigorous mathematical analysis of a nonlinear SSL model. The authors demonstrate that while supervised learning (SL) focuses strictly on label-related features, SSL acts as a "feature sponge," successfully capturing both label-related and "hidden" features that are often crucial for downstream task performance.
Methodology
The study analyzes a variant of the SimSiam architecture, a popular SSL framework that avoids model collapse through structural asymmetry. The model employs a 1-layer nonlinear encoder with sigmoid or tanh activation functions, trained using standard Gradient Descent (GD).
To bypass the complex, step-by-step iterative analysis typical of deep learning theory, the authors introduce a novel approach using the exact version of the Inverse Function Theorem. They first identify a local minimum for a simplified, "clean" version of the objective function and then use the theorem to prove that a corresponding local minimum exists for the actual, noisy objective used in training.
Data Distribution and Feature Capture
The researchers utilize a synthetic "toy" data distribution designed to isolate specific learning behaviors. The data is clustered in a high-dimensional space and contains two orthogonal unit-norm vectors: $e_1$ (the label-related feature) and $e_2$ (the hidden feature).
The core insight is that SSL, lacking labels to prioritize specific features, attempts to maximize similarity between different augmented views of the same data. This forces the learned weights to align with any consistent signal in the data, effectively capturing both $e_1$ and $e_2$. In contrast, SL is "lazy"; once it identifies $e_1$ well enough to satisfy classification margins, it has no mathematical incentive to learn $e_2$, treating it as noise.
Key Results and Simulation
The theoretical analysis proves that under a specific initialization region, the nonlinear SSL model is guaranteed to converge to a local minimum. In this state, the projection of both $e_1$ and $e_2$ onto the learned weight space is nearly 1, indicating that the model has successfully captured both features.
Simulation experiments with a 10-dimensional space confirm these findings. While the SL model's weights associated with the hidden feature $e_2$ were nearly zero, the SSL model maintained a high projection for $e_2$, as shown in the empirical distribution.
Conclusion and Future Work
This work provides a significant step toward a theoretical foundation for nonlinear SSL. By proving that SSL captures a more comprehensive set of features than SL, the authors offer a mathematical explanation for SSL's superior performance in transfer learning. Future work will focus on extending this "Inverse Function Theorem" analysis to multi-layer networks and more complex, hierarchical data distributions.
Summary updated