Redundancy Reduction for Efficient Representation Learning

39 concepts36 papers1 proposal

In deep learning, we often assume that increasing the dimensionality of a feature vector increases the model's capacity to represent information. However, recent research suggests that high-dimensional embeddings are frequently plagued by feature redundancy—a phenomenon where different dimensions of a representation become highly correlated, effectively collapsing the information into a much narrower subspace.

This redundancy, often termed "dimensional collapse" or "informational collapse," means that multiple neurons or embedding dimensions are encoding the same underlying signal rather than capturing distinct features. This inefficiency limits the model's expressive power, complicates interpretability, and hinders generalization.

The Problem of Dimensional Collapse

When a model is trained without explicit constraints on feature diversity, it tends to take the path of least resistance, focusing on a few dominant, highly discriminative patterns. Nguyen et al. describe this as a failure to utilize the model's full representational capacity, where the effective dimensionality is far lower than the actual vector size.

This issue is particularly acute in specific architectures and tasks:

  • Graph Neural Networks (GNNs): Repeated neighborhood aggregations cause node embeddings to become highly similar, creating a "feature space bottleneck" where standard weight-sharing cannot exploit the redundant signals (Sun et al.).
  • Self-Supervised Learning (SSL): In contrastive learning, samples can cluster too tightly around prototypes, leading to "prototype collapse" where internal diversity is lost (Mo et al.).
  • Transformers: Standard training objectives like next-token prediction do not inherently maximize information density. Hidden states often develop high correlation between dimensions, which Arefin et al. argue is a primary reason Transformers struggle with complex multi-step reasoning.

Decorrelation as a Regularization Strategy

To combat this, researchers have proposed various methods to explicitly "decorrelate" features, forcing the model to learn independent and diverse signals.

One of the most influential approaches is Barlow Twins, proposed by Zbontar et al.. Instead of using complex architectural asymmetries to avoid collapse, the authors introduce a loss function that makes the cross-correlation matrix of twin embeddings close to the identity matrix. By minimizing the off-diagonal terms, the model is forced to make the components of the embedding vector independent of each other, thereby reducing redundancy.

Similarly, Arefin et al. introduced Sequential Variance-Covariance Regularization (Seq-VCR) for Transformers. By penalizing the covariance between different dimensions of the hidden state, they maintain feature diversity throughout the layers, significantly improving the model's ability to perform internal calculations for tasks like multi-digit multiplication.

Orthonormalization and SVD

Another powerful mechanism for reducing redundancy is the use of Singular Value Decomposition (SVD) and orthonormalization. The intuition is that by forcing the basis of the feature space to be orthogonal, we ensure that each dimension captures a unique direction of variance.

Schuetze et al. propose a simple but effective "focal" representation by applying SVD twice with a normalization step in between. This process creates dimensions that are more semantically meaningful and independent. In the context of Graph Neural Networks, Henderson et al. use Batch Representation Orthonormalization (BRO) to disentangle node embeddings. They found that forcing features to be independent makes the model's predictions significantly more interpretable for medicinal chemists, as it prevents overlapping representations from obscuring the influence of specific input attributes.

Bio-inspired and Adversarial Perspectives

The drive for non-redundant features also appears in biological and competitive learning frameworks. Stephen Chung argues that standard cooperative learning leads to highly correlated features. As an alternative, Chung proposes a model where neurons act as self-interested agents competing to maximize their own weights, a strategy that naturally reduces statistical dependence and increases feature diversity.

In the realm of multi-model ensembles, Kang et al. address the "redundancy problem" where models trained on the same data converge on the same prominent features. Their framework, ACoRL, uses adversarial training to force new models to avoid features already captured by existing ones, creating an ensemble of "allies" that provide complementary rather than redundant information.

Why Redundancy Reduction Works

The success of these methods across diverse domains—from audio representation (Nguyen et al.) to medical image segmentation (Hassanpour et al.)—stems from a shared principle: information decoupling.

By stripping away redundant signals, the model is forced to spread its "attention" across a wider array of features. This leads to:

  1. Higher Efficiency: Models can achieve the same or better performance with fewer parameters or lower-dimensional embeddings (Deng et al.).
  2. Better Generalization: Decorrelated features are less likely to overfit to specific textures or source-domain biases (Asthana et al.).
  3. Improved Interpretability: When dimensions are independent, we can more clearly map specific features to specific model outputs (Henderson et al.).

Ultimately, the shift toward redundancy reduction represents a move away from simply building larger models toward building more information-dense ones.

Key Foundational Work

Two SVDs produce more focal deep learning representations (2013) — contributed 1 concept elements to this direction.
Color space normalization: Enhancing the discriminating power of color spaces for face recognition (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_feature_redundancy_representation_redund_c0bea35d,
  title = {feature redundancy (representation redundancy)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-redundancy-representation-redundancy/rdr_c0bea35ded3d4cd1a7bf3f4cf41f0d95},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_feature_redundancy_representation_redund_c0bea35d_v1,
  title = {feature redundancy (representation redundancy)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-redundancy-representation-redundancy/rdr_c0bea35ded3d4cd1a7bf3f4cf41f0d95/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake