Abstract
We establish local laws for sample covariance matrices $K = N^{-1}\sum_{i=1}^N \mathbf{g}_i\mathbf{g}_i^*$ where the random vectors $\mathbf{g}_1, \ldots, \mathbf{g}_N \in \mathbb{R}^n$ are independent with common covariance $\Sigma$. Previous work has largely focused on the separable model $\mathbf{g} = \Sigma^{1/2}\mathbf{w}$ with $\mathbf{w}$ having independent entries, but this structure is rarely present in statistical applications involving dependent or nonlinearly transformed data. Under a concentration assumption for quadratic forms $\mathbf{g}^*A\mathbf{g}$, we prove an optimal averaged local law showing that the Stieltjes transform of $K$ converges to its deterministic limit uniformly down to the optimal scale $\eta \geq N^{-1+\epsilon}$. Under an additional structural assumption on the cumulant tensors of $\mathbf{g}$ — which interpolates between the highly structured case of independent entries and generic dependence — we establish the full anisotropic local law, providing entrywise control of the resolvent $(K-zI)^{-1}$ in arbitrary directions. We discuss several classes of non-separable examples satisfying our assumptions, including conditionally mean-zero distributions, the random features model $\mathbf{g} = \sigma(X\mathbf{w})$ arising in machine learning, and Gaussian measures with nonlinear tilting. The proofs introduce a tensor network framework for analyzing fluctuation averaging in the presence of higher-order cumulant structure.
Introduction
Sample covariance matrices are foundational to statistics, physics, and machine learning. In the high-dimensional regime, where the number of features and samples grow proportionally, the global eigenvalue distribution is well-described by the Marchenko-Pastur law. However, understanding the spectrum at microscopic scales requires "local laws," which are essential for proving properties like eigenvalue rigidity and eigenvector delocalization.
Existing proofs for these local laws heavily rely on the separable model, where data vectors are linear transformations of independent random variables. In modern machine learning, data is rarely this simple; for instance, in neural networks or random features models, data undergoes nonlinear transformations (e.g., $\mathbf{g} = \sigma(X\mathbf{w})$), creating complex, non-separable dependencies. This paper addresses whether optimal local laws still hold for data distributions beyond this simple setting.
Methodology: Beyond Separability
To analyze non-separable data, the authors replace standard independence assumptions with two new structural conditions:
- Concentration of quadratic forms: An assumption that quadratic forms of the data concentrate tightly around their expected values.
- Cumulant tensor bounds: A structural assumption on higher-order cumulant tensors that interpolates between highly structured independent entries and generic, unstructured dependence.
To prove the full anisotropic local law, the authors introduce a tensor network framework. Proving local laws requires bounding massive, algebraically complex high-moment expansions. The authors map these expansions to a family of graph-theoretic tensor networks and employ a recursive "peeling" strategy—successively removing high-degree vertices and using cumulant assumptions to control the cost—until the network is reduced to simple chains and cycles that can be directly bounded.
Key Insights
- Quadratic Concentration and Averaged Laws: Strong concentration of quadratic forms is sufficient to prove the averaged local law, describing the Stieltjes transform of the matrix on average.
- Anisotropy and Cumulant Structure: The stronger anisotropic local law—required for proving eigenvector delocalization—necessitates specific decay structures in higher-order cumulants. Purely generic dependence is insufficient.
- Gaussian Equivalence in ML: For the random features model (a proxy for 2-layer neural networks), the paper proves a strong "Gaussian equivalence." At the microscopic spectral level, the resolvent of the nonlinear model is indistinguishable from a surrogate linear Gaussian model.
- Interdisciplinary Tools: Borrowing tensor network formalisms from quantum physics and constructive field theory provides a powerful method for managing the combinatorial explosion of high-moment expansions in random matrix theory.
Key Results
The study establishes several significant theoretical benchmarks:
- Optimal Averaged Local Law: Proved for non-separable sample covariance matrices down to the optimal microscopic scale ($\eta \ge N^{-1+\epsilon}$).
- Anisotropic Local Law: Established for the linearized resolvent under new cumulant assumptions.
- Eigenvector Delocalization: Proved as a corollary for both the sample covariance and Gram matrices in any fixed basis.
- Verification on ML Models: Theoretical assumptions were successfully verified on the random features model $\mathbf{g} = \sigma(X\mathbf{w})$ and Gaussian measures with nonlinear tilting.
Limitations and Future Directions
While the cumulant assumption accommodates compositional structures, verifying these bounds for deep neural networks remains a complex combinatorial challenge. Furthermore, the current proof method does not capture delicate fluctuation cancellations found in certain "disordered" physics systems, such as the spherical 4-spin glass model. Future work may explore the "characteristic flow method" (a dynamical approach using Dyson Brownian motion) as an alternative for analyzing non-separable random vectors.
Summary updated