Understanding Representation Learnability of Nonlinear Self-Supervised Learning

Ruofeng Yang, Xiangyuan Li, Bo Jiang, Shuai Li

Abstract: Self-supervised learning (SSL) has demonstrated remarkable success in learning data representations for downstream tasks. However, theoretical understanding of this process, particularly for nonlinear models, has remained limited. Most existing works treat nonlinear networks as "black boxes" or focus on linear simplifications. This paper provides the first accurate analysis of the learning results of a nonlinear SSL model. By considering a toy data distribution with label-related and hidden features, we prove that a 1-layer nonlinear SSL model trained via gradient descent converges to a local minimum that captures both feature types. In contrast, supervised learning (SL) models primarily capture label-related features. Our analysis utilizes the Inverse Function Theorem to describe the features learned at the local minimum, and we validate these findings through simulation experiments.

Introduction

Self-supervised learning has revolutionized fields like computer vision and natural language processing by enabling models to learn useful representations without explicit labels. Despite this empirical success, the theoretical underpinnings of why SSL learns richer representations than supervised learning are still being explored. A key question is "representation learnability": the ability of a model to capture latent data structures that might be irrelevant to a current task's labels but crucial for future downstream tasks.

Methods

The researchers analyzed a 1-layer nonlinear SSL model, specifically a variant of the SimSiam architecture. This model uses a nonlinear activation function (such as Sigmoid or Tanh) and is trained to maximize the similarity between different augmented views of the same data point.

The nonlinear SSL framework analyzed in the study, featuring a SimSiam-like architecture with data augmentation and a similarity-maximizing loss function.

To analyze this non-convex optimization problem, the authors introduced a representative "toy" data distribution. This distribution contains two orthogonal unit-norm features: $e_1$ (label-related) and $e_2$ (hidden). While $e_1$ distinguishes the classes in a supervised setting, $e_2$ represents information that is present in the data but not necessary for classification.

Theoretical Analysis

A major challenge in analyzing nonlinear models is the lack of closed-form solutions. The authors bypassed this by using the exact version of the Inverse Function Theorem. They first analyzed a simplified, noise-free objective function and then used the theorem as a bridge to prove that the local minima of the actual, noisy objective function still effectively capture the data features. They demonstrated that standard Gradient Descent (GD) converges to these "good" local minima at a geometric rate when initialized within a specific region.

Key Insights: SSL vs. Supervised Learning

The study highlights a fundamental difference in how SSL and SL interact with data features:

  1. SSL as a Feature Magnet: The SSL objective forces the model's weight matrix to align with all prominent features in the data. Because it seeks to maximize similarity between augmented views, any consistent structure (like $e_2$) becomes a target for learning.
  2. Supervised Learning's "Tunnel Vision": Supervised learning only needs to satisfy a margin constraint to separate classes. Once the model has captured enough of the label-related feature $e_1$ to achieve low classification error, it has no mathematical incentive to learn the hidden feature $e_2$.

Key Results and Simulations

The theoretical findings are summarized in two main theorems:

  • Theorem 1 (SSL Feature Capture): The SSL model captures both $e_1$ and $e_2$ almost perfectly. The projection of these features onto the learned weight space is shown to be nearly 1 ($|\Pi e| \geq 1 - O(\tau^3 d^{-1/2})$).
  • Theorem 2 (SL Feature Neglect): In the SL model, the weights associated with the hidden feature $e_2$ are shown to be vanishingly small as the data dimensionality $d$ increases, proving that SL effectively ignores the hidden feature.
Simulation results for the nonlinear SSL model: (a) the final weight matrix W, (b) learning curves for both features, and (c) projection histogram of the hidden feature.

Simulation experiments with 20 random seeds confirmed these theorems. As shown in the learning curves, the SSL model's weights for both $e_1$ and $e_2$ components grow and stabilize at non-zero values. In contrast, the authors' SL experiments showed that hidden feature components remained near zero throughout training.

Conclusion

This work provides a rigorous mathematical foundation for the superior transferability of SSL representations. By proving that nonlinear SSL models naturally capture hidden features that supervised models ignore, the authors offer a clear explanation for why SSL is so effective for diverse downstream tasks. Future work will aim to extend this analysis to deeper, multi-layer architectures and more complex, real-world data distributions.

Summary updated

Abstract

Self-supervised learning (SSL) has empirically shown its data representation learnability in many downstream tasks. There are only a few theoretical works on data representation learnability, and many of those focus on final data representation, treating the nonlinear neural network as a ``black box". However, the accurate learning results of neural networks are crucial for describing the data distribution features learned by SSL models. Our paper is the first to analyze the learning results of the nonlinear SSL model accurately. We consider a toy data distribution that contains two features: the label-related feature and the hidden feature. Unlike previous linear setting work that depends on closed-form solutions, we use the gradient descent algorithm to train a 1-layer nonlinear SSL model with a certain initialization region and prove that the model converges to a local minimum. Furthermore, different from the complex iterative analysis, we propose a new analysis process which uses the exact version of Inverse Function Theorem to accurately describe the features learned by the local minimum. With this local minimum, we prove that the nonlinear SSL model can capture the label-related feature and hidden feature at the same time. In contrast, the nonlinear supervised learning (SL) model can only learn the label-related feature. We also present the learning processes and results of the nonlinear SSL and SL model via simulation experiments.
BibTeX
@inproceedings{yang2023understanding,
  author = {Ruofeng Yang and Xiangyuan Li and Bo Jiang and Shuai Li},
  title = {{Understanding Representation Learnability of Nonlinear Self-Supervised Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=2YCW3TtIz6},
}
Figure