What Do We Maximize in Self-Supervised Learning And Why Does Generalization Emerge?

Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Yann LeCunICLR 2023 · OpenReview

Abstract

In this paper, we provide an information-theoretic (IT) understanding of self-supervised learning methods, their construction, and optimality. As a first step, we demonstrate how IT quantities can be obtained for deterministic networks, as an alternative to the commonly used unrealistic stochastic networks assumption. Secondly, we demonstrate how different SSL models can be (re)discovered based on first principles and highlight what the underlying assumptions of different SSL variants are. Third, we derive a novel generalization bound based on our IT understanding of SSL methods, providing generalization guarantees for the downstream supervised learning task. As a result of this bound, along with our unified view of SSL, we can compare the different approaches and provide general guidelines to practitioners. Consequently, our derivation and insights can contribute to a better understanding of SSL and transfer learning from a theoretical and practical perspective.
BibTeX
@inproceedings{shwartzziv2023what,
  author = {Ravid Shwartz-Ziv and Randall Balestriero and Kenji Kawaguchi and Yann LeCun},
  title = {{What Do We Maximize in Self-Supervised Learning And Why Does Generalization Emerge?}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=tuE-MnjN7DV},
}

Loading related content...

Figure