Abstract
In high-dimensional statistics, testing the independence of sub-vectors remains a significant challenge when the number of variables $p$ exceeds the sample size $n$. This paper introduces a test for the independence of $k$ high-dimensional sub-vectors of a normal vector. By defining a "block correlation matrix" and constructing a Schott-type statistic, we provide a robust testing framework that leverages Free Probability Theory and Random Matrix Theory. Unlike classical Likelihood Ratio Tests, this method does not require $n$ to be larger than $p$. Simulations and real-world analysis of S&P 500 stock returns demonstrate that the proposed statistic outperforms existing methods in both size and power.
Introduction
The "curse of dimensionality" poses a fundamental hurdle for classical statistical inference. In fields ranging from genomics to finance, researchers frequently encounter datasets where the number of variables ($p$) is significantly larger than the number of observations ($n$). Traditional tools, such as the Likelihood Ratio Test (LRT), rely on the invertibility of the sample covariance matrix. When $p > n$, this matrix becomes singular, causing the LRT to fail.
While previous research has attempted to adapt these tests for high-dimensional settings, many solutions impose strict constraints on the ratio of variables to samples. This paper addresses the problem by proposing a test of independence for $k$ sub-vectors that remains mathematically tractable and statistically powerful even in the most challenging high-dimensional environments.
Methodology: The Block Correlation Matrix
The core innovation of this research is the Block Correlation Matrix ($\mathbf{B}$). While a standard correlation matrix evaluates relationships between individual variables, the block correlation matrix extends this concept to groups of variables (sub-vectors).
The Schott-type Statistic
The authors define a test statistic, $s(\mathbf{B})$, based on the trace of the square of the off-diagonal blocks of this matrix. This approach is rooted in Canonical Correlation Analysis (CCA), where the sum of the eigenvalues of these blocks serves as a proxy for the total correlation between high-dimensional groups. This statistic is specifically designed to bypass the singularity issues that plague the LRT.
Theoretical Foundation: Free Probability
To determine the limiting distribution of the Schott-type statistic, the authors employ Free Probability Theory (FPT) and Random Matrix Theory (RMT). Specifically, they utilize the concept of "real second-order freeness" for Haar-distributed orthogonal matrices. This advanced mathematical framework allows for the description of eigenvalue fluctuations even as dimensions grow, providing a Central Limit Theorem (CLT) for the statistic that remains valid even when the sample size is small relative to the total dimension.
Experimental Results
The proposed method ($T_0$) was rigorously tested against two primary competitors: the renormalized LRT ($T_1$) and the trace criterion test ($T_2$).
Simulation Studies
Across four distinct scenarios varying mean vectors and covariance structures, the Schott-type statistic demonstrated:
- Robustness: It maintained an empirical size (false positive rate) near the 5% significance level, even in cases where $T_1$ and $T_2$ were mathematically inapplicable or performed poorly.
- Superior Power: $T_0$ consistently showed higher power (the ability to detect true dependence) than the trace criterion test ($T_2$) across various dimensions.
Case Study: S&P 500 Sector Interdependence
The authors applied the test to daily returns of 258 stocks from the S&P 500 during the last quarter of 2013 (64 trading days). The stocks were grouped into 11 sectors (e.g., Finance, Energy, Utilities) based on the North American Industry Classification System (NAICS).
- Sectoral Independence: At a 5% significance level, certain sectors, such as Finance, appeared relatively independent of others in specific pairings.
- Systemic Dependence: However, when testing groups of five or more sectors, the test revealed strong evidence of dependence. This illustrates the systemic nature of the stock market, where localized independence often masks broader structural interconnectedness.
Discussion and Conclusion
The Schott-type statistic offers a significant advancement in high-dimensional inference. Its primary advantages include:
- Singularity Avoidance: It does not require the calculation of determinants or matrix inverses, making it immune to the $p > n$ problem.
- Invariance: The test is invariant to the translation and scaling of variables, removing the need for prior knowledge of population means or variances.
Future Work
While the current framework assumes a multivariate normal distribution, the authors suggest that the test's performance on non-normal data remains an open area for research. Additionally, while the number of sub-vectors ($k$) is currently fixed, expanding the theory to allow $k$ to grow with $n$ represents a promising frontier. This work underscores the utility of Free Probability Theory—often viewed as an abstract branch of operator algebra—in solving practical statistical problems in the era of Big Data.
Summary updated