Redundancy Reduction via Feature Decorrelation

568 concepts346 papers1 proposal

A central challenge in self-supervised learning (SSL) is preventing representation collapse. When a neural network is trained to produce identical embeddings for two different augmented views of the same image, it often discovers a trivial shortcut: mapping every single input to the exact same constant vector.

To prevent this, traditional contrastive learning methods rely on negative pairs—explicitly forcing embeddings of different images to push away from each other. However, this requires large batch sizes or complex architectural constraints like stop-gradients and momentum encoders.

Feature decorrelation offers an elegant alternative. Instead of preventing collapse by comparing different instances across a batch, it prevents collapse by regularizing the features (dimensions) within the embedding space itself. By forcing the individual components of an embedding vector to be uncorrelated, the network is driven to maximize its information capacity, ensuring that every dimension learns something unique.

The Intuition: Fighting Dimensional Collapse

To understand why feature decorrelation works, consider a model that outputs a 1000-dimensional embedding vector. If every dimension changes in perfect lockstep with the others, the model's effective representation space is actually one-dimensional. This phenomenon is known as dimensional collapse. As analyzed by Tianyu Hua et al., dimensional collapse is a hidden performance killer where features become highly correlated and trapped on a low-dimensional manifold.

Feature decorrelation solves this by treating the embedding space as an information bottleneck. If we force the cross-correlation or covariance matrix of the embeddings to approximate an identity matrix, we enforce two critical properties:

  1. The Diagonal Elements $\to 1$ (Invariance & Variance): This ensures that each individual feature remains active and maintains high variance across the batch, preventing the entire representation from shrinking to zero.
  2. The Off-Diagonal Elements $\to 0$ (Decorrelation): This minimizes the redundancy between different features. If feature $A$ and feature $B$ have zero covariance, they cannot encode the same information.

By minimizing redundancy, the network is forced to distribute information across the entire vector, creating a "factorial code" where each coordinate captures an independent, non-redundant attribute of the data.

Key Frameworks

The strategy of using feature-level constraints to guide representation learning has been implemented through several prominent objectives.

Barlow Twins

In Barlow Twins: Self-Supervised Learning via Redundancy Reduction, Jure Zbontar et al. applied this principle by computing a cross-correlation matrix between the embeddings of two differently augmented views of the same data batch. By penalizing the off-diagonal elements of this matrix, the model learns noise-invariant features while ensuring that different dimensions do not capture overlapping information. This eliminates the need for negative samples entirely.

VICReg

Expanding on this concept, Adrien Bardes et al. introduced VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. Instead of a cross-correlation matrix, VICReg applies statistical constraints directly to the covariance matrix of each embedding branch individually. It explicitly decouples the objective into three terms: maintaining a minimum variance for each dimension, minimizing the distance between twin views (invariance), and driving the off-diagonal covariance elements to zero.

DeCov (Supervised Regularization)

While feature decorrelation has revolutionized self-supervised learning, its roots extend into supervised learning as a tool against overfitting. In Reducing Overfitting in Deep Networks by Decorrelating Representations Michael Cogswell, Faruk Ahmed et al. introduced DeCov, an auxiliary loss that penalizes the Frobenius norm of the covariance matrix of hidden activations (excluding the diagonal). The authors demonstrated that as deep networks overfit, internal correlations typically increase; explicitly forcing neurons to decorrelate serves as a powerful proxy for improving model generalization.

Why It Works: Information Maximization

Feature decorrelation works because it reframes representation learning from an instance-discrimination problem to an information-maximization problem.

When a model is trained with instance-level contrastive losses, it only cares about finding features that distinguish Image $A$ from Image $B$. If a single feature (like background color) is enough to separate them, the model has no incentive to learn other properties.

By contrast, a redundancy-reduction objective forces the model to utilize its entire dimensional capacity. If the model has 1000 dimensions available, the off-diagonal penalty prevents it from wasting 999 of them on redundant copies of the background color. To satisfy the loss, the network must find 1000 orthogonal axes of variation in the data—compelling it to capture subtle, diverse, and highly generalizable properties such as texture, shape, and structure.

Applications Beyond Computer Vision

Because feature decorrelation operates on the statistical properties of embedding spaces rather than data-specific structures, it generalizes seamlessly across domains:

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 373 papers
  2. Version 1 · 2026-10-04 · 417 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_feature_decorrelation_cross_correlation__1b45da23,
  title = {feature decorrelation (cross-correlation redundancy reduction)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-decorrelation-cross-correlation-redundancy-reduction/rdr_1b45da2360c74ecab46c7ccd4368ae1e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_feature_decorrelation_cross_correlation__1b45da23_v2,
  title = {feature decorrelation (cross-correlation redundancy reduction)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-decorrelation-cross-correlation-redundancy-reduction/rdr_1b45da2360c74ecab46c7ccd4368ae1e/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake