Abstract
Transfer learning plays a key role in advancing machine learning models, yet conventional supervised pretraining often undermines feature transferability by prioritizing features that minimize the pretraining loss. In this work, we adapt a self-supervised learning regularization technique from the VICReg method to supervised learning contexts, introducing Variance-Covariance Regularization (VCReg). This adaptation encourages the network to learn high-variance, low-covariance representations, promoting learning more diverse features. We outline best practices for an efficient implementation of our framework, including applying it to the intermediate representations. Through extensive empirical evaluation, we demonstrate that our method significantly enhances transfer learning for images and videos, achieving state-of-the-art performance across numerous tasks and datasets. VCReg also improves performance in scenarios like long-tail learning and hierarchical classification. Additionally, we show its effectiveness may stem from its success in addressing challenges like gradient starvation and neural collapse. In summary, VCReg offers a universally applicable regularization framework that significantly advances transfer learning and highlights the connection between gradient starvation, neural collapse, and feature transferability.
Introduction
Supervised pretraining is the standard foundation for many deep learning tasks. However, it suffers from a fundamental limitation: models often become overly specialized to the pretraining task. By focusing exclusively on the most discriminative features required to minimize training loss, they may ignore a wealth of information that is crucial for downstream transfer tasks.
This phenomenon manifests as two primary training pathologies:
- Gradient Starvation: The network identifies a few "easy" features and stops learning others, even if those additional features are vital for generalization.
- Neural Collapse: In the final stages of training, representations of the same class cluster so tightly that they lose internal diversity, making the model brittle when transferred to new domains.
VCReg aims to bridge this gap by incorporating principles from self-supervised learning into the supervised paradigm, fostering robust and transferable representations.
Methodology: Variance-Covariance Regularization
VCReg is adapted from the self-supervised VICReg method. While VICReg utilizes three terms—Variance, Invariance, and Covariance—the authors found that the "Invariance" term is redundant in supervised settings because class labels already enforce it.
VCReg focuses on two primary objectives:
- Variance Regularization: This term forces the standard deviation of each feature dimension to remain above a specific threshold, preventing "informational collapse" where neurons become inactive.
- Covariance Regularization: This term minimizes the off-diagonal elements of the covariance matrix, decorrelating features to ensure that different neurons capture non-redundant information.
A key innovation is applying VCReg to intermediate layers rather than just the final output. To maintain computational efficiency, the authors developed a fast implementation that modifies the backward pass directly, making the overhead comparable to standard Batch Normalization.
Experimental Evaluation and Insights
The authors validated VCReg across a wide range of benchmarks, including ImageNet-1K for image pretraining, Kinetics for video, and various transfer learning datasets like iNaturalist and Places205.
Key Insights
- Diversity Drives Transferability: By maintaining high-variance, low-covariance representations, VCReg preserves a richer set of features that standard supervised loss would typically discard.
- Mitigating Gradient Starvation: On synthetic tasks like the "Two-Moon" dataset, standard models often fail to learn complex curved boundaries once a simple linear separation is found. VCReg forces the model to continue learning, resulting in more robust decision margins.
Performance Highlights
- Superior Transfer Learning: On a ResNet-50 backbone, VCReg improved average transfer accuracy across nine datasets from 62.33% to 67.96%.
- Architecture Agnostic: The method provides consistent gains across ResNet, ConvNeXt, and Vision Transformers (ViT).
- Video and Robustness: VCReg boosted performance for state-of-the-art video models like VideoMAE and showed significantly higher resilience to Gaussian noise in input data.
Conclusion
VCReg offers a lightweight, plug-and-play regularization framework that significantly advances transfer learning. By addressing gradient starvation and neural collapse through intermediate representation diversity, it ensures that models remain "information-rich" and ready for diverse downstream applications. Future work may explore its synergy with other optimization techniques like Sharpness-Aware Minimization (SAM) and its efficacy in highly specialized domains like medical imaging.
Summary updated