Preventing Representation Collapse via Entropy Regularization

308 concepts224 papers1 proposal

The Gravity of Representation Collapse

When training deep neural networks, we often design objectives that encourage similarity. In self-supervised learning, we want different views of the same image to map close together. In deep regression, we want representations to align tightly with a continuous target value. In clustering, we want data points to group around cluster centroids.

However, these objectives suffer from a shared pathological failure mode: representation collapse. Left to its own devices, an optimization algorithm will find the easiest way to minimize these alignment losses. For a Siamese network, the easiest solution is to map every single input to a constant vector—a phenomenon known as complete mode collapse. For a regression model, standard losses like Mean Squared Error can compress the learned features into a tiny, low-entropy fraction of the available space.

To prevent this, traditional architectures rely on complex engineering workarounds: stop-gradients, momentum encoders, large negative-sample memory banks, or strict architectural constraints. But what if we could prevent collapse directly through the objective function itself?

Entropy as an Expansionary Force

The core intuition behind this research direction is to introduce an information-theoretic "counter-force" directly into the loss function: maximum entropy regularization.

If alignment losses act like gravity, pulling representations together, maximizing the entropy of the representation distribution acts like gas pressure, pushing them apart. Entropy measures the spread, diversity, or uncertainty of a distribution. By maximizing the marginal entropy of the learned features across a batch or dataset, we force the network to utilize the full capacity of its latent space.

This elegant formulation eliminates the need for specialized architectural tricks. For instance, in self-supervised learning, Xin Liu et al. developed Maximum Entropy Coding (MEC) in Self-Supervised Learning via Maximum Entropy Coding, demonstrating that explicitly maximizing feature entropy naturally prevents mode collapse without requiring momentum encoders or stop-gradients. Similarly, Xiasi Wang et al. proposed the Multi-View Entropy Bottleneck in MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck, balancing view alignment with differential entropy maximization to achieve highly robust representations without explicit negative samples.

Continuous Spaces and Differential Entropy

In continuous embedding spaces, maximizing entropy means spreading vectors uniformly across the available geometry (such as a unit hyper-sphere). Because computing the exact information entropy of a continuous distribution of high-dimensional vectors is intractable, researchers use differentiable proxy estimators.

One common approach is to maximize the distance between nearest neighbors. In Spreading vectors for similarity search, Alexandre Sablayrolles utilizes a differential entropy estimator as a loss function to force high-dimensional vectors to spread uniformly, maximizing indexing capacity. For image copy detection, Ed Pizzi et al. employ a Kozachenko-Leononenko differential entropy estimator in A Self-Supervised Descriptor for Image Copy Detection to push descriptors apart and prevent dimensional collapse.

Alternatively, variance and covariance constraints can serve as analytical proxies for entropy. As proven by Ravid Shwartz-Ziv et al. in An Information Theory Perspective on Variance-Invariance-Covariance Regularization, variance-covariance regularization techniques like VICReg are essentially implicit mutual information and entropy maximizers. By maintaining high differential entropy, models preserve crucial within-class variance rather than collapsing representations into single points, which Md Yousuf Harun et al. show significantly enhances out-of-distribution detection and transfer learning in Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning.

Discrete Spaces and the Entropy Balance

When dealing with discrete latent spaces—such as clustering layers, codebooks, or categorical assignments—the application of entropy requires a careful balancing act.

If we simply minimize the entropy of individual assignments, the model becomes highly confident but might assign every single data point to the exact same cluster (cluster collapse). To prevent this, the objective must optimize two distinct components of entropy:

  1. Minimize Conditional Entropy (Sharpness): Ensure that for any single input, the model's assignment to a specific cluster or attribute is highly certain and unambiguous.
  2. Maximize Marginal Entropy (Uniformity): Ensure that when averaged across the entire dataset, the distribution of cluster assignments is uniform, meaning every available cluster or codebook slot is utilized.

This dual-entropy mechanism is widely applicable. In COMPLETER: Incomplete Multi-View Clustering via Contrastive Prediction, Yijie Lin et al. incorporate entropy-based regularization to prevent multi-view clustering models from converging to trivial solutions where all data points belong to a single cluster. For model interpretability, Jayneel Parekh et al. demonstrate in A Framework to Learn with Interpretation that minimizing individual attribute entropy while maximizing global marginal entropy ensures that explanations are concise for individual samples yet diverse across the dictionary.

Cross-Disciplinary Impact

The principle of maximum entropy regularization extends far beyond self-supervised vision models, acting as a universal remedy for collapse across diverse machine learning domains:

By treating entropy not just as a metric to observe, but as an explicit optimization objective, modern architectures can maintain representation diversity, maximize latent capacity, and robustly avoid the trap of representation collapse.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cluster_collapse_prevention_193b694d,
  title = {cluster collapse prevention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cluster-collapse-prevention/rdr_193b694d418e42238fda78e25a58befd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_cluster_collapse_prevention_193b694d_v1,
  title = {cluster collapse prevention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cluster-collapse-prevention/rdr_193b694d418e42238fda78e25a58befd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake