Test of Independence for High-dimensional Random Vectors Based on Block Correlation Matrices

Zhigang Bao, Jiang Hu, Guangming Pan, Zhou WangarXiv 2014 · DOI · arXiv

Abstract

In high-dimensional statistics, testing the independence of sub-vectors remains a significant challenge when the number of variables $p$ exceeds the sample size $n$. This paper introduces a test for the independence of $k$ high-dimensional sub-vectors of a normal vector. By defining a "block correlation matrix" and constructing a Schott-type statistic, we provide a robust testing framework that leverages Free Probability Theory and Random Matrix Theory. Unlike classical Likelihood Ratio Tests, this method does not require $n$ to be larger than $p$. Simulations and real-world analysis of S&P 500 stock returns demonstrate that the proposed statistic outperforms existing methods in both size and power.

Introduction

The "curse of dimensionality" poses a fundamental hurdle for classical statistical inference. In fields ranging from genomics to finance, researchers frequently encounter datasets where the number of variables ($p$) is significantly larger than the number of observations ($n$). Traditional tools, such as the Likelihood Ratio Test (LRT), rely on the invertibility of the sample covariance matrix. When $p > n$, this matrix becomes singular, causing the LRT to fail.

While previous research has attempted to adapt these tests for high-dimensional settings, many solutions impose strict constraints on the ratio of variables to samples. This paper addresses the problem by proposing a test of independence for $k$ sub-vectors that remains mathematically tractable and statistically powerful even in the most challenging high-dimensional environments.

Methodology: The Block Correlation Matrix

The core innovation of this research is the Block Correlation Matrix ($\mathbf{B}$). While a standard correlation matrix evaluates relationships between individual variables, the block correlation matrix extends this concept to groups of variables (sub-vectors).

The Schott-type Statistic

The authors define a test statistic, $s(\mathbf{B})$, based on the trace of the square of the off-diagonal blocks of this matrix. This approach is rooted in Canonical Correlation Analysis (CCA), where the sum of the eigenvalues of these blocks serves as a proxy for the total correlation between high-dimensional groups. This statistic is specifically designed to bypass the singularity issues that plague the LRT.

Theoretical Foundation: Free Probability

To determine the limiting distribution of the Schott-type statistic, the authors employ Free Probability Theory (FPT) and Random Matrix Theory (RMT). Specifically, they utilize the concept of "real second-order freeness" for Haar-distributed orthogonal matrices. This advanced mathematical framework allows for the description of eigenvalue fluctuations even as dimensions grow, providing a Central Limit Theorem (CLT) for the statistic that remains valid even when the sample size is small relative to the total dimension.

Experimental Results

The proposed method ($T_0$) was rigorously tested against two primary competitors: the renormalized LRT ($T_1$) and the trace criterion test ($T_2$).

Simulation Studies

Across four distinct scenarios varying mean vectors and covariance structures, the Schott-type statistic demonstrated:

  • Robustness: It maintained an empirical size (false positive rate) near the 5% significance level, even in cases where $T_1$ and $T_2$ were mathematically inapplicable or performed poorly.
  • Superior Power: $T_0$ consistently showed higher power (the ability to detect true dependence) than the trace criterion test ($T_2$) across various dimensions.

Case Study: S&P 500 Sector Interdependence

The authors applied the test to daily returns of 258 stocks from the S&P 500 during the last quarter of 2013 (64 trading days). The stocks were grouped into 11 sectors (e.g., Finance, Energy, Utilities) based on the North American Industry Classification System (NAICS).

  • Sectoral Independence: At a 5% significance level, certain sectors, such as Finance, appeared relatively independent of others in specific pairings.
  • Systemic Dependence: However, when testing groups of five or more sectors, the test revealed strong evidence of dependence. This illustrates the systemic nature of the stock market, where localized independence often masks broader structural interconnectedness.

Discussion and Conclusion

The Schott-type statistic offers a significant advancement in high-dimensional inference. Its primary advantages include:

  1. Singularity Avoidance: It does not require the calculation of determinants or matrix inverses, making it immune to the $p > n$ problem.
  2. Invariance: The test is invariant to the translation and scaling of variables, removing the need for prior knowledge of population means or variances.

Future Work

While the current framework assumes a multivariate normal distribution, the authors suggest that the test's performance on non-normal data remains an open area for research. Additionally, while the number of sub-vectors ($k$) is currently fixed, expanding the theory to allow $k$ to grow with $n$ represents a promising frontier. This work underscores the utility of Free Probability Theory—often viewed as an abstract branch of operator algebra—in solving practical statistical problems in the era of Big Data.

Summary updated

Abstract

In this paper, we are concerned with the independence test for $k$ high-dimensional sub-vectors of a normal vector, with fixed positive integer $k$. A natural high-dimensional extension of the classical sample correlation matrix, namely block correlation matrix, is raised for this purpose. We then construct the so-called Schott type statistic as our test statistic, which turns out to be a particular linear spectral statistic of the block correlation matrix. Interestingly, the limiting behavior of the Schott type statistic can be figured out with the aid of the Free Probability Theory and the Random Matrix Theory. Specifically, we will bring the so-called real second order freeness for Haar distributed orthogonal matrices, derived in \cite{MP2013}, into the framework of this high-dimensional testing problem. Our test does not require the sample size to be larger than the total or any partial sum of the dimensions of the $k$ sub-vectors. Simulated results show the effect of the Schott type statistic, in contrast to those of the statistics proposed in \cite{JY2013} and \cite{JBZ2013}, is satisfactory. Real data analysis is also used to illustrate our method.
BibTeX
@misc{bao2014test,
  author = {Zhigang Bao and Jiang Hu and Guangming Pan and Zhou Wang},
  title = {{Test of Independence for High-dimensional Random Vectors Based on Block Correlation Matrices}},
  year = {2014},
  eprint = {1410.5082},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1410.5082},
}

Loading related content...

Figure