Abstract
Deep neural networks (DNNs) are often characterized by their ability to learn high-dimensional representations. However, this study reveals a counter-intuitive phenomenon: the rank of these representations—a proxy for information flow—diminishes monotonically and often exponentially as depth increases. We provide a theoretical proof for this "Principle of Rank Diminishing" and introduce computationally efficient tools to measure it across modern architectures. Our empirical findings on ImageNet-trained models demonstrate that this decay is universal, leading to a significant "independence deficit" where class predictions become highly redundant due to the low intrinsic dimensionality of the final feature space.
Introduction
While the success of deep learning is well-documented, the internal mechanics of how information is transformed across layers remain partially opaque. A key aspect of this transformation is the rank of the feature representations and the Jacobian matrices of the network's layers. Low-rank representations are often desirable for generalization, but an excessive loss of rank can lead to information bottlenecks. This paper investigates the layer-by-layer evolution of rank, providing both a rigorous theoretical framework and practical numerical metrics to quantify "rank diminishing" in state-of-the-art networks.
Theoretical Foundation and Numerical Metrics
The authors establish the Principle of Rank Diminishing, proving that the rank of a network's output relative to its input must decrease as it passes through successive layers. This is a direct consequence of the chain rule and the properties of matrix multiplication.
To measure this phenomenon in practice, the study introduces several innovations:
- Numerical Rank: A noise-robust metric that counts singular values exceeding a threshold relative to the spectral norm.
- Partial Jacobian Rank: A memory-efficient method to compute the rank of sub-matrices (e.g., from input patches), bypassing the massive VRAM requirements of full Jacobian matrices for deep networks like ResNet-50.
- Classification Dimension (ClsDim): A metric that estimates the intrinsic dimension of the final feature space by determining the minimum number of principal components needed to maintain classification performance.
Universal Rank Decay Across Architectures
Empirical validation on ImageNet shows that rank diminishing is a universal property of modern DNNs, including CNNs (ResNets), MLP-Mixers, and Transformers (ViT, Swin). Regardless of the specific architecture or initialization scheme, the rank of feature representations consistently drops as data moves deeper into the network.
The decay is often exponential. Interestingly, while standard layers like convolutions and fully connected layers contribute significantly to rank loss, normalization layers (such as BatchNorm and LayerNorm) appear to mitigate this effect, losing rank much more modestly.
The Independence Deficit
A striking consequence of rank diminishing is the Independence Deficit. Because the final feature space has a much lower intrinsic dimension than its ambient dimension (e.g., an intrinsic dimension of 131 for ResNet-50's 2048-unit space), different categories end up sharing representations.
Using Lasso regression, the authors demonstrate that a network's confidence in one category can often be linearly predicted from its confidence in a few others. For example, in a trained ResNet-50, the logit for "hamster" can be predicted with 98% accuracy using a linear combination of five other, often semantically unrelated, categories. This suggests that current training paradigms may induce non-realistic dependencies between classes.
Conclusion
Rank diminishing is an inherent impetus in deep neural networks, driven by both explicit architectural choices and implicit mathematical properties of coupled functions. While some rank loss is expected as networks extract high-level features, the extreme redundancy observed in the "independence deficit" highlights potential vulnerabilities and opportunities for architectural improvement. Future research into rank-preserving designs could lead to models with better generalization and more robust internal representations.
Summary updated