Multi-level Thresholding Test for High Dimensional Covariance Matrices

Song Xi Chen, Bin Guo, Yumou QiuarXiv 2019 · DOI · arXiv

Abstract

Testing the equality of two high-dimensional covariance matrices is a fundamental problem in modern statistics, with applications ranging from genomics to finance. This paper introduces the Multi-level Thresholding Test (MTT), a procedure designed to detect "rare and faint" differences—signals that are both sparse and individually weak. By developing a novel U-statistic composition and employing matrix blocking and coupling techniques, the authors establish the asymptotic distribution of the thresholding statistics. The MTT is shown to achieve the optimal minimax rate in signal strength, providing a powerful and theoretically grounded alternative to traditional $\ell_2$ and $\ell_{max}$ tests.

Introduction

In modern data science, researchers frequently encounter "large $p$, small $n$" problems, where the number of variables ($p$) far exceeds the number of observations ($n$). Comparing the covariance structures of two groups is vital for understanding how variables interact differently across populations.

Existing statistical tests generally fall into two categories:

  1. $\ell_2$-type tests: These aggregate squared differences across all entries. They are effective for "dense" signals (many small differences) but are often overwhelmed by noise when differences are sparse.
  2. $\ell_{max}$-type tests: These focus on the single largest difference. They excel in "sparse and strong" scenarios but fail when differences are individually faint.

The Multi-level Thresholding Test (MTT) addresses the "missing middle": the rare and faint regime, where differences are few and small.

Methodology

The core of the MTT is a thresholding statistic, $T_n(s)$, which sums only the entries of the difference matrix that exceed a specific threshold $s$. This approach effectively "mutes" the thousands of entries containing only noise, significantly increasing the signal-to-noise ratio.

Multi-level Approach

Since the true sparsity and strength of the signal are unknown a priori, the MTT adaptively considers a range of threshold levels. By evaluating multiple thresholds, the test identifies the most informative level for the specific dataset, ensuring high power across various signal configurations.

Technical Innovations

Proving the reliability of covariance tests is difficult because matrix entries are not independent. The authors overcome this using:

  • Matrix Blocking & Coupling: The matrix is partitioned into large blocks separated by small "buffer" strips. Using coupling techniques, these blocks are treated as nearly independent, allowing for rigorous mathematical proofs of the test's distribution.
  • U-statistic Representation: A novel U-statistic composition is used to establish the asymptotic Gumbel distribution of the thresholding statistics.
  • Bootstrap Calibration (MTT-BT): Because theoretical distributions can converge slowly in small samples, the authors provide a parametric bootstrap version. This ensures the test maintains a stable false-positive rate (size control) in practical applications.

Performance Evaluation

The paper evaluates the MTT using sophisticated simulations designed to mimic real-world data dependencies:

  • Design 1 (Auto-regressive): Simulates decaying correlations, typical of time-series data.
  • Design 2 (Block Diagonal): Simulates variables clustering into independent groups, such as functional gene networks.

Key Results

  • Superior Power: In simulations, the MTT-BT consistently outperformed benchmarks (like the SY, LC, and CLX tests) when signals were sparse. Unlike $\ell_{max}$ tests, the MTT captures the collective strength of multiple faint signals.
  • Minimax Optimality: The authors prove the MTT attains the minimax rate, meaning it reaches the theoretical limit of detection. It can identify differences at the lowest possible signal strength allowed by statistical theory.
  • Robustness: The test maintained its performance across both Gaussian and non-normal (Gamma) distributions, demonstrating robustness to distributional assumptions.

Conclusion and Future Work

The Multi-level Thresholding Test provides a rigorous framework for high-dimensional covariance testing, filling a critical gap in detecting sparse and faint signals. While the bootstrap procedure ensures accuracy, it introduces computational costs in extremely high dimensions. Future research may explore more efficient computational methods and extend the thresholding framework to scale-invariant correlation matrix testing.

Summary updated

Abstract

We consider testing the equality of two high-dimensional covariance matrices by carrying out a multi-level thresholding procedure, which is designed to detect sparse and faint differences between the covariances. A novel U-statistic composition is developed to establish the asymptotic distribution of the thresholding statistics in conjunction with the matrix blocking and the coupling techniques. We propose a multi-thresholding test that is shown to be powerful in detecting sparse and weak differences between two covariance matrices. The test is shown to have attractive detection boundary and to attain the optimal minimax rate in the signal strength under different regimes of high dimensionality and the sparsity of the signal. Simulation studies are conducted to demonstrate the utility of the proposed test.
BibTeX
@misc{chen2019multi,
  author = {Song Xi Chen and Bin Guo and Yumou Qiu},
  title = {{Multi-level Thresholding Test for High Dimensional Covariance Matrices}},
  year = {2019},
  eprint = {1910.13074},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.13074},
}
Figure