Variance-Covariance Regularization Improves Representation Learning

Jiachen Zhu, Katrina Evtimova, Yubei Chen, Ravid Shwartz-Ziv, Yann LeCunSubmission to ICLR 2025 · Rejected

TL;DR

Supervised pretraining often encourages neural networks to take "shortcuts," capturing only the minimal features needed to minimize training loss at the expense of generalizability. This paper introduces Variance-Covariance Regularization (VCReg)—an adaptation of the self-supervised VICReg objective—into supervised training across intermediate layers. VCReg enforces high-variance and decorrelated feature representations, substantially improving transfer learning performance across standard image and video benchmarks while adding virtually no computational overhead.

What Problem Does This Paper Address?

In standard supervised pretraining, models are prone to two common failure modes:

  1. Gradient starvation: The network prioritizes easy-to-learn features and stops learning additional, complementary features.
  2. Neural collapse: During late training stages, representations of the same class collapse toward their mean vector, drastically diminishing feature diversity.

While these representations succeed at classifying the source dataset, they generalize poorly when transferred to downstream tasks. Although self-supervised learning (SSL) uses explicit anti-collapse criteria, supervised pipelines typically lack mechanisms to prevent features from becoming overly redundant or compressed.

Methods

VCReg adapts the variance and covariance penalties from VICReg (Bardes et al., 2021) while dropping the invariance objective, as explicit class labels already guide task semantics:

  • Variance Loss ($\ell_{\text{var}}$): Penalizes feature dimensions whose standard deviation drops below a threshold, preventing representation collapse.
  • Covariance Loss ($\ell_{\text{cov}}$): Penalizes off-diagonal elements of the covariance matrix to minimize correlation and redundancy between latent dimensions.
  • Intermediate Layer Application: Rather than regularizing only the penultimate embedding, VCReg is applied after each network block. To handle spatial feature maps efficiently without inflating covariance matrices, spatial positions within a feature map are treated as individual samples.
  • Smooth L1 Penalty: Because spatial locations share correlations, large covariance outliers can cause gradient instability. The squared covariance term is replaced with a Smooth L1 loss.
  • Fast Custom Autograd Implementation: Because VCReg does not alter forward activations, the authors bypass standard autograd computation of the loss graph and directly inject analytic regularizer gradients into the backward pass. This reduces execution latency by over $5\times$ compared to naive implementations, matching the overhead of standard Batch Normalization layers.
       [ Input Features ]
              │
              ├──► [ Standard Forward Pass ] ──► Supervised Loss
              │
     (Per-Block Representations)
              │
    Treat spatial locations as samples
    Compute Feature Covariance Matrix
              │
     ┌────────┴────────┐
     ▼                 ▼
Variance Term    Covariance Term
(Keep std ≥ 1)   (Minimize off-diagonals via Smooth L1)
     └────────┬────────┘
              ▼
   Direct Backward Gradient Injection (Fast)

Datasets

  • Pretraining Datasets: ImageNet-1k (images); Kinetics-400 and Kinetics-710 (videos).
  • Transfer Evaluation (Images, Linear Probing): iNaturalist 2018, Places205, Food-101, Stanford Cars, FGVC Aircraft, Oxford-IIIT Pets, Oxford 102 Flowers, and Describable Textures Dataset (DTD).
  • Transfer Evaluation (Video Action Recognition): HMDB51 (including noisy Gaussian-corrupted variants).
  • Specialized Benchmarks: CIFAR-10-LT and CIFAR-100-LT (long-tailed imbalance with ratio 100); CIFAR-100 and 5 ImageNet subsets (living_9, mixed_10, mixed_13, geirhos_16, big_12) for hierarchical/subclass probing; synthetic Two-Moon dataset.

(No new datasets were introduced; all benchmarks are standard community datasets.)

Key Insights and Ideas

  • Regularizing Intermediate Layers Matters Most: Applying VCReg strictly to the final layer offers limited gains. Enforcing decorrelation and variance across early and middle stages forces the model to construct richer intermediate abstraction hierarchies.
  • Invariance Loss is Redundant in Supervised Regimes: VICReg relies heavily on invariance between augmented views, but the authors find that supervised cross-entropy handles class invariance natively. Retaining only variance and covariance is sufficient to diversify representations.
  • Measurable Reduction in Neural Collapse: Quantified via Class-Distance Normalized Variance (CDNV) and Nearest Class-Center (NCC) metrics, VCReg prevents features from collapsing into narrow geometric points, preserving substantially more mutual information between the input and latent representations.

Key Results

  • Image Transfer Learning: Incorporating VCReg yields consistent gains across backbones:
  • ResNet-50: Average transfer accuracy increases from 62.33% to 67.96% (+5.63%), outperforming competing diversity regularizers like DeCov (62.83%) and WLD-Reg (63.63%).
  • ConvNeXt-Tiny: Average transfer accuracy improves from 72.44% to 73.40%.
  • ViT-Base-32: Improves from 64.20% to 65.19%.
  • Video Transfer Learning: Pretrained video transformers (VideoMAE, VideoMAEv2, ViViT) fine-tuned on HMDB51 see consistent gains of +0.4% to +0.8%, and show markedly improved resilience to Gaussian input noise.
  • Hierarchical Classification & Long-Tail: When trained only on coarse superclasses, ConvNeXt with VCReg improves CIFAR-100 subclass probing from 60.7% to 72.9% (+12.2%). On CIFAR-100-LT (imbalance ratio 100), ResNet-32 accuracy improves from 37.4% to 40.4%.
  • Computational Efficiency: On an A100 GPU, the fast backward implementation processes ConvNeXt-T in 0.471s per batch (vs. 2.951s for naive VCReg and 0.468s for Batch Normalization).

Limitations and Future Work

  • Compute-Constrained Hyperparameter Tuning: Due to heavy compute requirements (~92,000 GPU hours total), the authors tuned only the loss weights ($\alpha, \beta$) and relied on default PyTorch training recipes for all standard hyperparameters.
  • Data Domain Scope: Pretraining experiments were restricted to standard natural images (ImageNet) and videos (Kinetics). Evaluating transferability across disparate domains (e.g., medical imaging, satellite data, or non-visual modalities) remains open for exploration.

Summary updated

Abstract

Transfer learning plays a key role in advancing machine learning models, yet conventional supervised pretraining often undermines feature transferability by prioritizing features that minimize the pretraining loss. In this work, we adapt a self-supervised learning regularization technique from the VICReg method to supervised learning contexts, introducing Variance-Covariance Regularization (VCReg). This adaptation encourages the network to learn high-variance, low-covariance representations, promoting learning more diverse features. We outline best practices for an efficient implementation of our framework, including applying it to the intermediate representations. Through extensive empirical evaluation, we demonstrate that our method significantly enhances transfer learning for images and videos, achieving state-of-the-art performance across numerous tasks and datasets. VCReg also improves performance in scenarios like long-tail learning and hierarchical classification. Additionally, we show its effectiveness may stem from its success in addressing challenges like gradient starvation and neural collapse. In summary, VCReg offers a universally applicable regularization framework that significantly advances transfer learning and highlights the connection between gradient starvation, neural collapse, and feature transferability.
BibTeX
@misc{zhu2023variance,
  author = {Jiachen Zhu and Katrina Evtimova and Yubei Chen and Ravid Shwartz-Ziv and Yann LeCun},
  title = {{Variance-Covariance Regularization Improves Representation Learning}},
  year = {2023},
  eprint = {2306.13292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.13292},
}

Loading related content...

Figure