Variance-Covariance Regularization Improves Representation Learning
Its authors withdrew this paper, so there is no full text to read or summarize.
Lists
Only you see this
Abstract
Transfer learning plays a key role in advancing machine learning models, yet conventional supervised pretraining often undermines feature transferability by prioritizing features that minimize the pretraining loss. Recent progress in self-supervised learning (SSL) has introduced regularization techniques that bolster feature transferability. In this work, we adapt an SSL regularization technique from the VICReg method to supervised learning contexts, introducing Variance-Covariance Regularization (VCReg). This adaptation encourages the network to learn a high-variance, low-covariance representation, promoting the learning of more diverse features. We outline best practices for implementing this regularization framework into various neural network architectures and present an optimized strategy for regularizing intermediate representations. Through extensive empirical evaluation, we demonstrate that our method significantly enhances transfer learning, achieving excellent performance across numerous tasks and datasets. VCReg also improves performance in scenarios like long-tail learning, and hierarchical classification. Additionally, we conduct analyses to suggest that its effectiveness may stem from its success in addressing challenges like gradient starvation and neural collapse. In summary, VCReg offers a universally applicable regularization framework that significantly advances the state of transfer learning, highlights the connection between gradient starvation, neural collapse, and feature transferability, and potentially opens a new avenue for regularization in this domain.
BibTeX
@misc{zhu2023variance,
author = {Jiachen Zhu and Ravid Shwartz-Ziv and Yubei Chen and Yann LeCun},
title = {{Variance-Covariance Regularization Improves Representation Learning}},
year = {2023},
eprint = {2306.13292},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2306.13292},
}Loading related content...