The Gravity of Representation Collapse
When training deep neural networks, we often design objectives that encourage similarity. In self-supervised learning, we want different views of the same image to map close together. In deep regression, we want representations to align tightly with a continuous target value. In clustering, we want data points to group around cluster centroids.
However, these objectives suffer from a shared pathological failure mode: representation collapse. Left to its own devices, an optimization algorithm will find the easiest way to minimize these alignment losses. For a Siamese network, the easiest solution is to map every single input to a constant vector—a phenomenon known as complete mode collapse. For a regression model, standard losses like Mean Squared Error can compress the learned features into a tiny, low-entropy fraction of the available space.
To prevent this, traditional architectures rely on complex engineering workarounds: stop-gradients, momentum encoders, large negative-sample memory banks, or strict architectural constraints. But what if we could prevent collapse directly through the objective function itself?
Entropy as an Expansionary Force
The core intuition behind this research direction is to introduce an information-theoretic "counter-force" directly into the loss function: maximum entropy regularization.
If alignment losses act like gravity, pulling representations together, maximizing the entropy of the representation distribution acts like gas pressure, pushing them apart. Entropy measures the spread, diversity, or uncertainty of a distribution. By maximizing the marginal entropy of the learned features across a batch or dataset, we force the network to utilize the full capacity of its latent space.
This elegant formulation eliminates the need for specialized architectural tricks. For instance, in self-supervised learning, Xin Liu et al. developed Maximum Entropy Coding (MEC) in Self-Supervised Learning via Maximum Entropy Coding, demonstrating that explicitly maximizing feature entropy naturally prevents mode collapse without requiring momentum encoders or stop-gradients. Similarly, Xiasi Wang et al. proposed the Multi-View Entropy Bottleneck in MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck, balancing view alignment with differential entropy maximization to achieve highly robust representations without explicit negative samples.
Continuous Spaces and Differential Entropy
In continuous embedding spaces, maximizing entropy means spreading vectors uniformly across the available geometry (such as a unit hyper-sphere). Because computing the exact information entropy of a continuous distribution of high-dimensional vectors is intractable, researchers use differentiable proxy estimators.
One common approach is to maximize the distance between nearest neighbors. In Spreading vectors for similarity search, Alexandre Sablayrolles utilizes a differential entropy estimator as a loss function to force high-dimensional vectors to spread uniformly, maximizing indexing capacity. For image copy detection, Ed Pizzi et al. employ a Kozachenko-Leononenko differential entropy estimator in A Self-Supervised Descriptor for Image Copy Detection to push descriptors apart and prevent dimensional collapse.
Alternatively, variance and covariance constraints can serve as analytical proxies for entropy. As proven by Ravid Shwartz-Ziv et al. in An Information Theory Perspective on Variance-Invariance-Covariance Regularization, variance-covariance regularization techniques like VICReg are essentially implicit mutual information and entropy maximizers. By maintaining high differential entropy, models preserve crucial within-class variance rather than collapsing representations into single points, which Md Yousuf Harun et al. show significantly enhances out-of-distribution detection and transfer learning in Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning.
Discrete Spaces and the Entropy Balance
When dealing with discrete latent spaces—such as clustering layers, codebooks, or categorical assignments—the application of entropy requires a careful balancing act.
If we simply minimize the entropy of individual assignments, the model becomes highly confident but might assign every single data point to the exact same cluster (cluster collapse). To prevent this, the objective must optimize two distinct components of entropy:
- Minimize Conditional Entropy (Sharpness): Ensure that for any single input, the model's assignment to a specific cluster or attribute is highly certain and unambiguous.
- Maximize Marginal Entropy (Uniformity): Ensure that when averaged across the entire dataset, the distribution of cluster assignments is uniform, meaning every available cluster or codebook slot is utilized.
This dual-entropy mechanism is widely applicable. In COMPLETER: Incomplete Multi-View Clustering via Contrastive Prediction, Yijie Lin et al. incorporate entropy-based regularization to prevent multi-view clustering models from converging to trivial solutions where all data points belong to a single cluster. For model interpretability, Jayneel Parekh et al. demonstrate in A Framework to Learn with Interpretation that minimizing individual attribute entropy while maximizing global marginal entropy ensures that explanations are concise for individual samples yet diverse across the dictionary.
Cross-Disciplinary Impact
The principle of maximum entropy regularization extends far beyond self-supervised vision models, acting as a universal remedy for collapse across diverse machine learning domains:
- Deep Regression: Standard regression objectives often lead to collapsed feature spaces. In Improving Deep Regression with Ordinal Entropy, Shihao Zhang et al. introduce an Ordinal Entropy regularizer that emulates the high-entropy spread of classification tasks, significantly boosting performance in depth estimation and age prediction.
- Federated Learning: To mitigate "activation divergence" where local models become overly specialized to local data distributions, Wei Chen et al. propose FedMAX in FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning, maximizing the entropy of activation vectors to ensure more universal feature representations.
- Reinforcement Learning: In non-reconstructive latent models, Vincent Francois-Lavet shows in Combined Reinforcement Learning via Abstract Representations that an entropy-maximizing loss function forces distinct environmental states into different regions of a bounded abstract space, allowing meaningful internal maps to emerge even in reward-free environments.
By treating entropy not just as a metric to observe, but as an explicit optimization objective, modern architectures can maintain representation diversity, maximize latent capacity, and robustly avoid the trap of representation collapse.