On singular values of data matrices with general independent columns

Tianxing Mei, Chen Wang, Jianfeng YaoThe Annals of Statistics 2023 · DOI · arXiv

Abstract

In high-dimensional statistics, understanding the singular values of large data matrices is essential for signal detection and dimensionality reduction. This paper establishes a limiting spectral distribution (LSD) for a $p \times n$ data matrix $X_n$ with independent columns that may follow different distributions. By assuming that the column covariance matrices are asymptotically simultaneously diagonalizable, we provide a unified framework that encompasses several classical models, including weighted sample covariance and Gram matrices. We further apply this theory to multi-dimensional diffusion processes, matrix-valued auto-regressive models, and generalized finite mixture models.

1. Introduction

The analysis of large-scale data matrices where the number of features $p$ is comparable to the number of observations $n$ is a cornerstone of modern statistics. A fundamental tool in this field is the Marčenko-Pastur law, which describes the asymptotic behavior of eigenvalues for sample covariance matrices. However, the standard Marčenko-Pastur framework relies on the assumption that observations are independent and identically distributed (i.i.d.).

In practice, data often exhibit heteroscedasticity or time-varying structures. For instance, financial returns often show volatility clustering, and genomic data may contain samples from subpopulations with varying noise levels. This paper, "On singular values of data matrices with general independent columns" by Mei, Wang, and Yao, addresses these complexities by establishing a unified mathematical framework for matrices with independent but non-identically distributed columns.

2. The General Model and Assumptions

The researchers consider a $p \times n$ matrix $X_n = (x_{n1}, \dots, x_{nn})$, where each column $x_{nj}$ is an independent $p$-dimensional vector. The covariance of each column is denoted by $\Sigma_{nj} = \text{Cov}(x_{nj})$.

The primary challenge in analyzing such a matrix is the potential for each column to have a unique covariance structure. To make the problem tractable, the authors introduce the assumption of Asymptotic Simultaneous Diagonalizability. This suggests that while the individual covariance matrices $\Sigma_{nj}$ differ, they can be approximately diagonalized by a common unitary matrix.

To describe the spectra of these covariances, the authors utilize a link function $f(a, b)$. Here, $a$ represents row-level parameters and $b$ represents column-level parameters. This functional approach allows for a flexible representation of how variance scales across both dimensions of the data matrix.

3. Methodology: The Stieltjes Transform

The core mathematical machinery used is the Stieltjes transform method from Random Matrix Theory (RMT). The Stieltjes transform $s(z)$ of a distribution is a complex-valued function that encodes information about the distribution's moments. By deriving a system of functional equations for the Stieltjes transform, the authors can uniquely determine the Limiting Spectral Distribution (LSD) as $p, n \to \infty$ such that $p/n \to c > 0$.

Furthermore, the authors invoke the Lindeberg principle to establish universality. This ensures that the resulting LSD depends primarily on the first two moments of the data (assuming the existence of finite fourth moments), rather than the specific underlying distribution (e.g., Gaussian vs. non-Gaussian).

4. Key Theoretical Results

The central contribution is Theorem 2.1, which proves the existence and uniqueness of the LSD for the singular values of $X_n$. This theorem provides the functional equations necessary to compute the density of the singular values for a wide array of matrix models.

This result is significant because it unifies several previously disparate models:

  • Marčenko-Pastur Law: The classic i.i.d. case.
  • Weighted Sample Covariance: Where columns are scaled by different weights.
  • Gram Matrices: Used extensively in kernel methods and machine learning.

5. Applications

The paper demonstrates the versatility of the framework through three primary applications:

5.1 Multi-dimensional Diffusion Processes

In finance, the "realized covariance matrix" is used to estimate risk from high-frequency data. Previous models often assumed "isotropic" volatility, where all assets scale similarly over time. This paper successfully derives the LSD for anisotropic cases, allowing for complex, time-varying interactions between different assets with unique volatility trends.

5.2 Matrix-valued Auto-Regressive (MAR) Models

MAR models are increasingly popular in macroeconomics for modeling interdependencies between multiple variables over time. The authors derive the LSD for the singular values of data matrices generated by these models, showing a near-perfect match between theoretical density curves and empirical simulations in numerical experiments.

5.3 Generalized Finite Mixture Models

When data is collected from a population consisting of several distinct subgroups, it is modeled as a finite mixture. The authors provide the LSD for such models, which is crucial for understanding the performance of clustering and classification algorithms in high-dimensional settings.

6. Conclusion and Limitations

This work provides a robust mathematical foundation for analyzing complex, high-dimensional datasets where the i.i.d. assumption fails. However, some constraints remain. The model assumes column independence, which may not hold in ultra-high-frequency financial data where short-term dependencies exist. Additionally, the assumption of simultaneous diagonalizability, while common, remains a specific structural constraint. Future research may look to relax these assumptions or explore the convergence rates for finite-sample sizes to better inform practical applications in data science.

Summary updated

Abstract

We analyze the singular values of a large p×n data matrix Xn=(xn1,…,xnn), where the columns {xnj} are independent p-dimensional vectors, possibly with different distributions. Assuming that the covariance matrices Σnj=Cov(xnj) of the column vectors can be asymptotically simultaneously diagonalized, with appropriately converging spectra, we establish a limiting spectral distribution (LSD) for the singular values of Xn when both dimensions p and n grow to infinity in comparable magnitudes. Our matrix model goes beyond and includes many different types of sample covariance matrices in existing work, such as weighted sample covariance matrices, Gram matrices, and sample covariance matrices of a linear time series model. Furthermore, three applications of our general approach are developed. First, we obtain the existence and uniqueness of the LSD for realized covariance matrices of a multi-dimensional diffusion process with anisotropic time-varying co-volatility. Second, we derive the LSD for singular values of data matrices from a recent matrix-valued auto-regressive model. Finally, we also obtain the LSD for singular values of data matrices from a generalized finite mixture model.
BibTeX
@article{mei2023singular,
  author = {Tianxing Mei and Chen Wang and Jianfeng Yao},
  title = {{On singular values of data matrices with general independent columns}},
  journal = {The Annals of Statistics},
  year = {2023},
  doi = {10.1214/23-aos2263},
}
Figure