VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution

Jaeill Kim, Suhyun Kang, Duhun Hwang, Jungwook Shin, Wonjong Rhee

Abstract

This paper introduces von Neumann Entropy (VNE) as a versatile, plug-and-play regularizer for deep learning representations. By directly manipulating the eigenvalue distribution of the representation's autocorrelation matrix, VNE improves performance across diverse tasks—including domain generalization, meta-learning, and self-supervised learning—while avoiding the "dead neuron" pitfalls of traditional methods. We demonstrate that VNE provides a theoretically grounded approach to controlling representation rank, isotropy, and disentanglement, offering a flexible tool for optimizing feature quality across various learning paradigms.

Introduction

In deep learning, the quality of learned representations—the features extracted by a model—is often a more significant determinant of performance than the specific architecture employed. Ideally, these representations should exhibit properties such as decorrelation (to reduce redundancy), high rank (to utilize available dimensions), and isotropy (uniformity in all directions).

Traditionally, researchers have enforced these properties by regularizing the Frobenius norm of the autocorrelation matrix, attempting to force it toward an identity matrix. However, this approach is often unstable and prone to causing "dead units"—neurons that become permanently inactive. Furthermore, different tasks require different representation "shapes." For instance, while meta-learning benefits from high-rank features, domain generalization may require compressing representations to ignore domain-specific noise.

Methods: von Neumann Entropy Regularization

The authors propose regularizing the von Neumann entropy (VNE) of the representation. Borrowed from quantum information theory, VNE is calculated as the Shannon entropy of the eigenvalues ($\lambda_j$) of the representation's autocorrelation matrix ($C_{auto}$):

$$S(C_{auto}) = -\sum \lambda_j \log \lambda_j$$

VNE is integrated as a simple penalty term in the loss function: $\mathcal{L}_{total} = \mathcal{L}_{task} - \alpha \cdot S(C_{auto})$.

Advantages over Frobenius Norm

  1. Eigenvalue Focus: VNE directly targets the spectral distribution rather than individual matrix elements.
  2. Stability: It handles zero eigenvalues gracefully ($0 \log 0 = 0$), preventing the collapse of dimensions and the "dead neuron" problem where traditional methods fail to recover dimensions that have collapsed to zero.
  3. Flexibility: By adjusting the sign of the regularization coefficient ($\alpha$), one can either maximize entropy ($VNE^+$) to encourage high-rank, isotropic features, or minimize it ($VNE^-$) to compress the representation.

Experimental Evaluation

The authors evaluated VNE across a wide array of benchmarks, including domain generalization (PACS, VLCS, OfficeHome, TerraIncognita), meta-learning (mini-ImageNet), self-supervised learning (ImageNet-100/1K), and generative modeling (CIFAR-10).

Key Results

The experimental results highlight the broad utility of VNE across different learning paradigms:

  • Domain Generalization (DG): Interestingly, DG tasks perform better with $VNE^-$. This suggests that for generalization across environments, models benefit from discarding spurious, domain-specific features and focusing on a smaller set of invariant features. As shown in Figure 1(a), $VNE^-$ improved the state-of-the-art SWAD algorithm across multiple datasets (e.g., +0.9% on OfficeHome).
  • Meta-Learning: $VNE^+$ improved six popular algorithms, including MAML and ProtoNet. For ProtoNet 1-shot, it boosted accuracy on mini-ImageNet by 3.65% (Figure 1(b)).
  • Self-Supervised Learning (SSL): The proposed I-VNE+ method avoids "dimensional collapse" without requiring complex mechanisms like momentum encoders or stop-gradients, achieving 86.3% Top-1 accuracy on ImageNet-100 and superior Average Precision in transfer tasks (Figure 1(c)).
  • Generative Models (GANs): VNE improved the Frechet Inception Distance (FID) across seven architectures, notably reducing DCGAN's FID from 42.55 to 35.44 (Figure 1(d)).
Performance improvements across four distinct tasks using VNE regularization: (a) Domain Generalization, (b) Meta-learning, (c) Self-supervised learning, and (d) GANs.

Discussion and Limitations

While VNE is a powerful tool, it introduces a hyperparameter $\alpha$ that requires tuning. Additionally, because the autocorrelation matrix is estimated over mini-batches, performance may be sensitive to very small batch sizes. Theoretically, the paper establishes that maximizing VNE is mathematically equivalent to maximizing rank, disentanglement, and isotropy under specific conditions. Future work may further explore why spectral-domain regularization proves more robust than direct feature-space entropy maximization.

Summary updated

Abstract

Since the introduction of deep learning, a wide scope of representation properties, such as decorrelation, whitening, disentanglement, rank, isotropy, and mutual information, have been studied to improve the quality of representation. However, manipulating such properties can be challenging in terms of implementational effectiveness and general applicability. To address these limitations, we propose to regularize von Neumann entropy~(VNE) of representation. First, we demonstrate that the mathematical formulation of VNE is superior in effectively manipulating the eigenvalues of the representation autocorrelation matrix. Then, we demonstrate that it is widely applicable in improving state-of-the-art algorithms or popular benchmark algorithms by investigating domain-generalization, meta-learning, self-supervised learning, and generative models. In addition, we formally establish theoretical connections with rank, disentanglement, and isotropy of representation. Finally, we provide discussions on the dimension control of VNE and the relationship with Shannon entropy. Code is available at: https://github.com/jaeill/CVPR23-VNE.
BibTeX
@inproceedings{kim2023vne,
  author = {Jaeill Kim and Suhyun Kang and Duhun Hwang and Jungwook Shin and Wonjong Rhee},
  title = {{VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure