Canonical correlation coefficients of high-dimensional normal vectors: finite rank case

Zhigang Bao, Jiang Hu, Guangming Pan, Zhou WangarXiv 2014 · DOI · arXiv

TL;DR

This paper characterizes a "phase transition" in high-dimensional Canonical Correlation Analysis (CCA). When the number of features ($p, q$) is proportional to the sample size ($n$), sample correlations are heavily biased; the authors prove that true population correlations can only be detected if they exceed a specific mathematical threshold ($r_c$), providing a formula to recover the true signal from the noisy sample data.

What Problem Does This Paper Address?

Canonical Correlation Analysis (CCA) is a classic tool used to find the maximum correlation between two sets of variables (e.g., relating genetic markers to clinical outcomes). In the "classical" regime, where you have many more observations than variables ($n \gg p, q$), sample correlations are reliable estimates of true population correlations.

However, in modern "high-dimensional" data, $p$ and $q$ are often as large as $n$. In this scenario, the sample canonical correlation matrix becomes "noisy." Even if two sets of variables are completely independent, the sample correlations will not be zero; they spread out into a "bulk" distribution (known as the Wachter distribution). The challenge is determining when a real correlation is strong enough to "pop out" of this noise floor and how to accurately estimate its true value once it does.

Methods

The authors employ Random Matrix Theory (RMT) to analyze the eigenvalues of the sample canonical correlation matrix. Their approach treats the problem as a finite-rank perturbation of the "null case" (where variables are independent).

Key mathematical tools include:

  • Stieltjes Transforms: Used to study the limiting spectral distribution of the matrices.
  • Free Probability Theory: Specifically the R-transform, which allows the authors to calculate how the distribution of eigenvalues shifts when two independent random matrices are added or multiplied.
  • Determinant Equations: They derive a master equation (the "location lemma") to pinpoint exactly where "outlier" eigenvalues (the signals) will land relative to the "bulk" (the noise).

Datasets

The paper is a theoretical mathematical proof and does not use real-world empirical datasets. Instead, it uses simulated data generated from multivariate normal distributions to validate the theorems.

  • Simulated Setup: For example, Figure 1 demonstrates a case with $n=5000$, $p=500$, and $q=1000$, showing how predicted outliers align perfectly with simulated sample eigenvalues.

Key Insights and Ideas

  • The Threshold Effect ($r_c$): The most critical insight is that a population correlation $r_i$ must be strictly greater than a threshold $r_c$ (defined by the ratios $p/n$ and $q/n$) to be detectable. If $r_i \leq r_c$, the sample correlation will simply disappear into the right edge of the noise bulk.
  • Signal Inflation: When a signal is detected ($r_i > r_c$), the sample correlation $\lambda_i$ is not equal to the true correlation $r_i$. It is a biased "outlier."
  • The Recovery Formula: The authors provide an explicit mapping: $\gamma_i = f(r_i, c_1, c_2)$. Because this mapping is invertible, researchers can take a measured sample correlation and "calculate back" to find the true underlying population correlation.
  • Independence of the Bulk: The global "shape" of the noise (the bulk) remains unchanged by a few strong correlations; only the extreme top eigenvalues are affected.

Key Results

The paper culminates in Theorem 1.8, which defines the almost sure limits of the sample canonical correlation coefficients ($\lambda_i$):

  1. Above Threshold: If the true squared correlation $r_i > r_c$, the sample squared correlation $\lambda_i$ converges to: $$\gamma_i = r_i(1 - c_1 + c_1 r_i^{-1})(1 - c_2 + c_2 r_i^{-1})$$ where $c_1 = p/n$ and $c_2 = q/n$.

  2. Below Threshold: If $r_i \leq r_c$, the sample squared correlation $\lambda_i$ simply sticks to the edge of the noise bulk ($d_r$).

  3. Phase Transition: The threshold $r_c$ is explicitly defined as a function of the dimensionality ratios $c_1$ and $c_2$, providing a clear boundary for signal detection in high-dimensional settings.

Limitations and Future Work

  • Distributional Assumptions: The proofs rely on the vectors being normally distributed (Gaussian). While the authors suggest the results might hold for more general distributions (the "universality" property in RMT), this is not formally proven here.
  • Fixed Rank: The study assumes the number of true correlations ($k$) is fixed and does not grow with the sample size $n$.
  • Fluctuation Analysis: The paper focuses on where the eigenvalues land (first-order limits). It does not delve into the "fluctuations" (the distribution around those limits, such as Tracy-Widom limits), which would be necessary for constructing formal hypothesis tests or confidence intervals.

Summary updated

Abstract

Consider a normal vector $\mathbf{z}=(\mathbf{x}',\mathbf{y}')'$, consisting of two sub-vectors $\mathbf{x}$ and $\mathbf{y}$ with dimensions $p$ and $q$ respectively. With $n$ independent observations of $\mathbf{z}$ at hand, we study the correlation between $\mathbf{x}$ and $\mathbf{y}$, from the perspective of the Canonical Correlation Analysis, under the high-dimensional setting: both $p$ and $q$ are proportional to the sample size $n$. In this paper, we focus on the case that $Σ_{\mathbf{x}\mathbf{y}}$ is of finite rank $k$, i.e. there are $k$ nonzero canonical correlation coefficients, whose squares are denoted by $r_1\geq\cdots\geq r_k>0$. Under the additional assumptions $(p+q)/n\to y\in (0,1)$ and $p/q\not\to 1$, we study the sample counterparts of $r_i,i=1,\ldots,k$, i.e. the largest k eigenvalues of the sample canonical correlation matrix $S_{\mathbf{x}\mathbf{x}}^{-1}S_{\mathbf{x}\mathbf{y}}S_{\mathbf{y}\mathbf{y}}^{-1}S_{\mathbf{y}\mathbf{x}}$, namely $λ_1\geq\cdots\geq λ_k$. We show that there exists a threshold $r_c\in(0,1)$, such that for each $i\in\{1,\ldots,k\}$, when $r_i\leq r_c$, $λ_i$ converges almost surely to the right edge of the limiting spectral distribution of the sample canonical correlation matrix, denoted by $d_r$. When $r_i>r_c$, $λ_i$ possesses an almost sure limit in $(d_r,1]$, from which we can recover $r_i$ in turn, thus provide an estimate of the latter in the high-dimensional scenario.
BibTeX
@misc{bao2014canonical,
  author = {Zhigang Bao and Jiang Hu and Guangming Pan and Zhou Wang},
  title = {{Canonical correlation coefficients of high-dimensional normal vectors: finite rank case}},
  year = {2014},
  eprint = {1407.7194},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1407.7194},
}
Figure