Abstract
We show how the replica method can be used to compute the asymptotic eigenvalue spectrum of a real Wishart product matrix. This provides a compact, elementary derivation of a polynomial condition on the Stieltjes transform first proved by Müller [IEEE Trans. Inf. Theory. 48, 2002].
Introduction
In Random Matrix Theory (RMT), Wishart matrices—formed by the product of a random matrix and its transpose—are foundational. However, the study of product matrices, resulting from the multiplication of multiple independent random matrices, presents significant mathematical challenges. These matrices are central to the study of Deep Linear Networks in machine learning, where the Gram matrix of a network with Gaussian weights is a Wishart product matrix. Understanding its eigenvalue spectrum is critical for predicting signal propagation and generalization performance. While the complex-valued case has been addressed using advanced algebraic techniques, this paper provides a compact and elementary derivation for the real-valued case, focusing on both the overall density and the extreme eigenvalues.
Methodology: The Replica Method
The authors utilize the replica method, a technique from the statistical physics of disordered systems, to bypass complex combinatorial proofs. The approach involves:
- Stieltjes Transform: Mapping the problem to a partition function of a physical system using the Edwards-Jones approach.
- The Replica Trick: Calculating the $m$-th moment of the partition function and taking the limit $m \to 0$ to average the logarithm of the partition function.
- Saddle Point Analysis: Assuming replica-symmetric solutions to find the dominant contributions in the large-matrix limit.
- Extreme Eigenvalues as Ground States: Interpreting the maximum and minimum eigenvalues as ground-state energies of a spherical spin glass at zero temperature.
Results: Eigenvalue Distribution and Extremes
The theoretical derivation successfully re-derives Müller’s polynomial equation for the Stieltjes transform, which characterizes the entire eigenvalue distribution. Numerical simulations with matrix sizes up to $n_0 = 2048$ demonstrate excellent agreement with these analytical results.
A key contribution is the derivation of explicit polynomial conditions for the spectral edges, $\lambda_{max}$ and $\lambda_{min}$. For the case of equal relative widths $\alpha$ across layers, these conditions simplify to quadratic equations. The results confirm that if any intermediate layer is narrower than the input ($\alpha < 1$), the minimum eigenvalue vanishes, indicating a loss of rank in the deep network.
Discussion and Future Directions
The study highlights the synergy between physics-based heuristics and rigorous random matrix theory, showing that the replica method offers a direct path to universal results. Notably, the conditions for extreme eigenvalues in real Wishart product matrices match those of the complex case, suggesting a high degree of universality.
However, the current work is limited to linear networks and Gaussian weight distributions. Future research should extend this replica-based approach to non-linear activation functions (e.g., ReLU) and non-Gaussian weights. Additionally, exploring the double-scaling limit where both depth and width tend to infinity remains an important open question for understanding very deep architectures.
Summary updated