Representation Learning via Redundancy Reduction and Decorrelation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2417 papers568 conceptsfrozen 2026-10-04

In self-supervised learning (SSL), the primary challenge is avoiding representation collapse. Without labels, a neural network can "cheat" by mapping every input to the same constant vector, achieving perfect similarity between different views of the same image while learning absolutely nothing.

While early methods like SimCLR relied on "negative samples" (explicitly pushing different images apart) to prevent this, a growing body of research suggests a more elegant solution: treating the embedding space as a communication channel and minimizing the redundancy between its dimensions.

The Barlow Twins Objective

The foundational work in this direction is Barlow Twins, introduced by Jure Zbontar, Ishan Misra, Yann LeCun, and colleagues. The core idea is to take two augmented views of a batch of data, pass them through a network, and compute a cross-correlation matrix between their outputs.

The objective function then tries to make this matrix as close to the identity matrix as possible. This simple goal accomplishes two things simultaneously:

  1. Invariance (The Diagonal): By forcing the diagonal elements toward 1, the model ensures that the same component in the two twin embeddings is highly correlated. This makes the representation robust to noise and augmentations.
  2. Redundancy Reduction (The Off-Diagonal): By forcing the off-diagonal elements toward 0, the model decorrelates the different dimensions of the embedding vector.

This second point is the "secret sauce." If dimension $i$ and dimension $j$ are uncorrelated, they must be capturing different information. This forces the network to spread information across the entire vector rather than collapsing into a single, redundant signal.

Why Decorrelation Prevents Collapse

To understand why this works, consider the "lazy neuron" problem. In standard training, multiple neurons often co-adapt to detect the same dominant feature (like "is there a dog?"). Michael Cogswell and Faruk Ahmed addressed this with DeCov Reducing Overfitting in Deep Networks by Decorrelating Representations, showing that penalizing cross-covariance forces neurons to learn diverse, non-redundant features.

In the context of SSL, redundancy reduction acts as a pressure that pushes the representation to fill the available "volume" of the embedding space. If the model tries to collapse all inputs to a single point, the variance of each dimension drops, and the correlations between dimensions spike—violating the identity matrix target. As Manh Tuan Do and Kijung Shin note in their work on Graph Neural Networks, this increases the intrinsic dimensionality of the learned features, ensuring the model utilizes its full expressive capacity.

Generalizing Across Domains

The intuition of redundancy reduction has proven remarkably versatile, moving far beyond simple image classification:

  • Graphs: Piotr Bielak and Tomasz Kajdanowicz adapted this to Graph Barlow Twins, removing the need for complex graph augmentations or negative samples. Similarly, Yue Liu and Xinwang Liu developed DCRN, which reduces redundancy at both the sample and feature levels to prevent "lazy" clustering.
  • Reinforcement Learning: In SIM, Keyu Wu and Zhenghua Chen use redundancy reduction to create intrinsic rewards. By filtering out redundant environmental noise, agents focus on the most informative features of their surroundings.
  • Knowledge Distillation: Lingyu Si and colleagues demonstrated in CMKD that a student model can learn better from a teacher by mimicking the teacher's "dimensional structure"—specifically, its channel-wise decorrelation patterns.

Beyond Linear Decorrelation

While the original Barlow Twins uses a linear cross-correlation measure, recent work has pushed toward capturing more complex dependencies.

Pierre-François De Plaen and Luc Van Gool introduced ADMin Adversarial Dependence Minimization, which uses an adversarial "dependency predictor" to eliminate nonlinear redundancies that simple correlation matrices might miss. Similarly, Chuang Niu and Ge Wang explored IMSVD, which uses information-theoretic measures like joint entropy to minimize redundancy, allowing for high performance even with very small embedding dimensions.

The Intuition of the Factorial Code

The ultimate goal of these methods is to produce a factorial code: a representation where each dimension is statistically independent of the others.

When a model achieves this, it has effectively "disentangled" the input. Instead of one giant, messy signal, the representation becomes a collection of independent "rationales" Jiangmeng Li. As Ryan Henderson showed with Batch Representation Orthonormalization, this independence doesn't just improve accuracy—it makes the model's internal logic more interpretable to humans, as each feature dimension is forced to stand on its own.

Papers in this version

Showing 60 of 417 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 373 papers
  2. Version 1 · 2026-10-04 · 417 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_feature_decorrelation_cross_correlation__1b45da23,
  title = {feature decorrelation (cross-correlation redundancy reduction)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-decorrelation-cross-correlation-redundancy-reduction/rdr_1b45da2360c74ecab46c7ccd4368ae1e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_representation_learning_via_redundancy_r_1b45da23_v1,
  title = {Representation Learning via Redundancy Reduction and Decorrelation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representation-learning-via-redundancy-reduction-and-decorrelation/rdr_1b45da2360c74ecab46c7ccd4368ae1e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}