Two-Sample Tests for High Dimensional Means with Thresholding and Data Transformation

Song Xi Chen, Jun Li, Ping‐Shou ZhongarXiv 2014 · DOI · arXiv

Abstract

We consider testing for two-sample means of high dimensional populations by thresholding. Two tests are investigated, which are designed for better power performance when the two population mean vectors differ only in sparsely populated coordinates. The first test is constructed by carrying out thresholding to remove the non-signal bearing dimensions. The second test combines data transformation via the precision matrix with the thresholding. The benefits of the thresholding and the data transformations are showed by a reduced variance of the test thresholding statistics, the improved power and a wider detection region of the tests. Simulation experiments and an empirical study are performed to confirm the theoretical findings and to demonstrate the practical implementations.

Introduction

In the modern era of data science, researchers frequently encounter the "large $p$, small $n$" problem, where the number of variables ($p$) vastly exceeds the number of observations ($n$). This is particularly common in fields like genomics, where thousands of gene expressions are measured across a small cohort of patients. Traditional statistical tools, such as Hotelling’s $T^2$ test, are inapplicable in this regime because they require the inversion of a covariance matrix that becomes singular when $p > n$.

While several high-dimensional tests have been developed to address this—most notably the Chen-Qin (2010) test—they often rely on $L_2$ norms that aggregate signals across all dimensions. These methods are highly effective when differences between populations are "dense" (spread across many variables). However, in many biological and financial contexts, differences are sparse, meaning only a tiny fraction of variables actually differ between groups. In these cases, the noise from thousands of non-differing dimensions can overwhelm the signal, leading to a significant loss of statistical power.

The Challenge of Sparse Signals

The primary hurdle in sparse high-dimensional testing is not necessarily that the signal is too small, but that the cumulative variance of the null dimensions is too large. When we sum the differences across all $p$ dimensions, the "non-signal bearing" dimensions contribute noise that dilutes the "needle in the haystack."

Chen, Li, and Zhong propose a framework to overcome this by focusing exclusively on the dimensions that matter. By integrating thresholding techniques with data transformation, their approach effectively lowers the "detection boundary," allowing researchers to identify subtle differences that were previously statistically invisible.

Methodology

1. Thresholding and Multi-Level Selection

The authors introduce a thresholding mechanism to filter out noise. Instead of a simple summation, the test statistic only includes dimensions where the difference between means exceeds a specific threshold. This process removes the variance contributed by dimensions with no signal. Because the optimal threshold is rarely known a priori, the authors utilize a multi-level thresholding approach (similar to "Higher Criticism"). This allows the test to adaptively search across a range of thresholds, capturing both a few strong signals or a larger number of faint signals.

2. Data Transformation via the Precision Matrix

Thresholding is most effective when variables are independent. However, real-world data often exhibits complex correlations. To address this, the authors employ a data transformation using the precision matrix (the inverse of the covariance matrix). This transformation "rotates" the data into a space where variables are nearly independent, enhancing the signal-to-noise ratio.

To estimate the precision matrix in high dimensions, the paper utilizes a banded Cholesky decomposition. This assumes a natural ordering of variables (such as genomic position on a chromosome) where correlations decay as the distance between variables increases.

Key Insights

  • Variance Reduction: The core benefit of thresholding is the drastic reduction in the variance of the test statistic. By discarding dimensions that do not contribute to the mean difference, the test becomes much more sensitive.
  • Lowering the Detection Boundary: The authors provide theoretical proof that combining transformation with thresholding expands the "detection region." This means the test can distinguish between populations even when the signal strength is lower than what is required by standard $L_2$ or max-norm tests.
  • Bootstrap Calibration: High-dimensional test statistics often converge slowly to their theoretical distributions (like the Gumbel distribution), which can lead to an inflated rate of false positives. The authors implement a parametric bootstrap method to calibrate the test, ensuring more accurate p-values in practical applications.

Empirical Results

Simulation Studies

Through extensive simulations using Gaussian and Gamma distributions, the authors compared their transformed multi-thresholding test (Mult2) against the standard Chen-Qin (CQ) test and max-norm tests (CLX). The results consistently showed that Mult2 maintained superior power in sparse settings while remaining robust across different signal strengths ($r$) and sparsity levels ($\beta$).

Human Breast Cancer Analysis

The practical utility of the method was demonstrated using a microarray gene expression dataset from Richardson et al. (2006). The study compared sporadic basal-like cancers (BLC) against non-BLC specimens. While the standard CQ test failed to find significance in several Gene Ontology (GO) terms, the thresholding-based test identified significant differences in categories such as "signal transduction" and "metal ion binding." These findings suggest that the new framework can uncover biological insights that aggregate-sum tests overlook.

Conclusion and Future Directions

The proposed thresholding and transformation framework represents a significant advancement in high-dimensional hypothesis testing. By intelligently filtering noise and decorrelating variables, it provides a robust solution for the sparse-signal problems prevalent in modern data science.

Future research may focus on relaxing the "bandable" covariance assumption. While genomic data often has a natural ordering, other types of data do not. Exploring alternative precision matrix estimators, such as CLIME or graphical LASSO, could extend the reach of this powerful testing framework to even broader classes of high-dimensional data.

Summary updated

Abstract

We consider testing for two-sample means of high dimensional populations by thresholding. Two tests are investigated, which are designed for better power performance when the two population mean vectors differ only in sparsely populated coordinates. The first test is constructed by carrying out thresholding to remove the non-signal bearing dimensions. The second test combines data transformation via the precision matrix with the thresholding. The benefits of the thresholding and the data transformations are showed by a reduced variance of the test thresholding statistics, the improved power and a wider detection region of the tests. Simulation experiments and an empirical study are performed to confirm the theoretical findings and to demonstrate the practical implementations.
BibTeX
@misc{chen2014two,
  author = {Song Xi Chen and Jun Li and Ping‐Shou Zhong},
  title = {{Two-Sample Tests for High Dimensional Means with Thresholding and Data Transformation}},
  year = {2014},
  eprint = {1410.2848},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1410.2848},
}

Loading related content...

Figure