Abstract
This paper introduces von Neumann Entropy (VNE) as a versatile, plug-and-play regularizer for deep learning representations. By directly manipulating the eigenvalue distribution of the representation's autocorrelation matrix, VNE improves performance across diverse tasks—including domain generalization, meta-learning, and self-supervised learning—while avoiding the "dead neuron" pitfalls of traditional methods. We demonstrate that VNE provides a theoretically grounded approach to controlling representation rank, isotropy, and disentanglement, offering a flexible tool for optimizing feature quality across various learning paradigms.
Introduction
In deep learning, the quality of learned representations—the features extracted by a model—is often a more significant determinant of performance than the specific architecture employed. Ideally, these representations should exhibit properties such as decorrelation (to reduce redundancy), high rank (to utilize available dimensions), and isotropy (uniformity in all directions).
Traditionally, researchers have enforced these properties by regularizing the Frobenius norm of the autocorrelation matrix, attempting to force it toward an identity matrix. However, this approach is often unstable and prone to causing "dead units"—neurons that become permanently inactive. Furthermore, different tasks require different representation "shapes." For instance, while meta-learning benefits from high-rank features, domain generalization may require compressing representations to ignore domain-specific noise.
Methods: von Neumann Entropy Regularization
The authors propose regularizing the von Neumann entropy (VNE) of the representation. Borrowed from quantum information theory, VNE is calculated as the Shannon entropy of the eigenvalues ($\lambda_j$) of the representation's autocorrelation matrix ($C_{auto}$):
$$S(C_{auto}) = -\sum \lambda_j \log \lambda_j$$
VNE is integrated as a simple penalty term in the loss function: $\mathcal{L}_{total} = \mathcal{L}_{task} - \alpha \cdot S(C_{auto})$.
Advantages over Frobenius Norm
- Eigenvalue Focus: VNE directly targets the spectral distribution rather than individual matrix elements.
- Stability: It handles zero eigenvalues gracefully ($0 \log 0 = 0$), preventing the collapse of dimensions and the "dead neuron" problem where traditional methods fail to recover dimensions that have collapsed to zero.
- Flexibility: By adjusting the sign of the regularization coefficient ($\alpha$), one can either maximize entropy ($VNE^+$) to encourage high-rank, isotropic features, or minimize it ($VNE^-$) to compress the representation.
Experimental Evaluation
The authors evaluated VNE across a wide array of benchmarks, including domain generalization (PACS, VLCS, OfficeHome, TerraIncognita), meta-learning (mini-ImageNet), self-supervised learning (ImageNet-100/1K), and generative modeling (CIFAR-10).
Key Results
The experimental results highlight the broad utility of VNE across different learning paradigms:
- Domain Generalization (DG): Interestingly, DG tasks perform better with $VNE^-$. This suggests that for generalization across environments, models benefit from discarding spurious, domain-specific features and focusing on a smaller set of invariant features. As shown in Figure 1(a), $VNE^-$ improved the state-of-the-art SWAD algorithm across multiple datasets (e.g., +0.9% on OfficeHome).
- Meta-Learning: $VNE^+$ improved six popular algorithms, including MAML and ProtoNet. For ProtoNet 1-shot, it boosted accuracy on mini-ImageNet by 3.65% (Figure 1(b)).
- Self-Supervised Learning (SSL): The proposed I-VNE+ method avoids "dimensional collapse" without requiring complex mechanisms like momentum encoders or stop-gradients, achieving 86.3% Top-1 accuracy on ImageNet-100 and superior Average Precision in transfer tasks (Figure 1(c)).
- Generative Models (GANs): VNE improved the Frechet Inception Distance (FID) across seven architectures, notably reducing DCGAN's FID from 42.55 to 35.44 (Figure 1(d)).
Discussion and Limitations
While VNE is a powerful tool, it introduces a hyperparameter $\alpha$ that requires tuning. Additionally, because the autocorrelation matrix is estimated over mini-batches, performance may be sensitive to very small batch sizes. Theoretically, the paper establishes that maximizing VNE is mathematically equivalent to maximizing rank, disentanglement, and isotropy under specific conditions. Future work may further explore why spectral-domain regularization proves more robust than direct feature-space entropy maximization.
Summary updated