Data-adaptive statistics for multiple hypothesis testing in high-dimensional settings

Weixin Cai, Nima S. Hejazi, Alan HubbardarXiv 2017 · DOI · arXiv

Abstract

Current statistical inference problems in areas like astronomy, genomics, and marketing routinely involve the simultaneous testing of thousands -- even millions -- of null hypotheses. For high-dimensional multivariate distributions, these hypotheses may concern a wide range of parameters, with complex and unknown dependence structures among variables. In analyzing such hypothesis testing procedures, gains in efficiency and power can be achieved by performing variable reduction on the set of hypotheses prior to testing. We present in this paper an approach using data-adaptive multiple testing that serves exactly this purpose. This approach applies data mining techniques to screen the full set of covariates on equally sized partitions of the whole sample via cross-validation. This generalized screening procedure is used to create average ranks for covariates, which are then used to generate a reduced (sub)set of hypotheses, from which we compute test statistics that are subsequently subjected to standard multiple testing corrections. The principal advantage of this methodology lies in its providing valid statistical inference without the \textit{a priori} specifying which hypotheses will be tested. Here, we present the theoretical details of this approach, confirm its validity via a simulation study, and exemplify its use by applying it to the analysis of data on microRNA differential expression.

Introduction: The Multiplicity Problem in High-Dimensional Data

In modern scientific research, particularly in fields like genomics and high-frequency marketing, researchers often face the challenge of identifying meaningful signals within massive datasets. When testing millions of hypotheses simultaneously, standard statistical corrections—designed to prevent false positives—often become overly conservative. This "multiplicity problem" can act as a "power killer," where real biological or economic signals are buried under the weight of corrections like Benjamini-Hochberg (BH).

Traditionally, researchers avoid this by pre-specifying a small subset of variables to test based on prior knowledge. However, such knowledge is often unavailable. Using the same data to both discover promising variables and test them leads to circular reasoning and biased results. To address this, Cai, Hejazi, and Hubbard (2017) proposed a data-adaptive multiple testing framework that integrates data mining and formal inference.

Methodology: Data-Adaptive Screening via Cross-Validation

The proposed method utilizes a generalized screening procedure built on $V$-fold cross-validation to maintain a strict separation between discovery and inference. The workflow follows these steps:

  1. Data Splitting: The dataset is divided into $V$ equal partitions.
  2. Screening (Discovery Phase): For each fold, the algorithm uses the remaining $V-1$ folds to rank all variables using data mining techniques, such as ranking by effect size.
  3. Estimation (Inference Phase): Test statistics are calculated for only the top-ranked variables using the held-out fold.
  4. Aggregation and Correction: Estimates are averaged across all folds to create a data-adaptive target parameter. Finally, standard False Discovery Rate (FDR) corrections (like Benjamini-Hochberg) are applied to this significantly reduced subset of high-potential hypotheses.

This approach allows the data to guide the selection of hypotheses without introducing the bias typically associated with "p-hacking."

Key Results and Empirical Validation

The authors validated their methodology using both simulated and real-world datasets.

Simulation Study

In a simulation involving $1,000,000$ potential biomarkers with only 10 true effects, the data-adaptive method demonstrated superior performance. While standard BH corrections failed to identify 80% of the true signals, the adaptive approach recovered 70% of them as statistically significant.

Distribution of adjusted p-values across one million covariates, highlighting the isolation of significant signals.

The effectiveness of the screening process is further evidenced by the stability and significance of the top-ranked covariates. As shown in the table below, the method consistently identifies variables with extremely low adjusted p-values and high cross-validation stability.

Top 15 covariates identified by the data-adaptive procedure in the simulation study.

microRNA Assay

In a study of 5,639 human microRNA expressions from individuals exposed to benzene, standard corrections yielded zero significant findings. By contrast, the data-adaptive approach—reducing the set to the top 30 candidates—identified 39 significant miRNAs (19 down-regulated and 20 up-regulated) that were previously undetectable.

Conclusion and Future Directions

The data-adaptive framework bridges the gap between exploratory data mining and rigorous statistical inference. By adaptively reducing the hypothesis space, researchers can significantly boost statistical power in high-dimensional settings.

Future work may explore the use of more sophisticated machine learning algorithms in the screening phase to capture non-linear relationships. While computationally more intensive than single-pass tests, the increasing availability of parallel computing and the release of the data.adapt.multi.test R package make this methodology highly accessible for modern "big data" challenges.

Summary updated

Abstract

Current statistical inference problems in areas like astronomy, genomics, and marketing routinely involve the simultaneous testing of thousands -- even millions -- of null hypotheses. For high-dimensional multivariate distributions, these hypotheses may concern a wide range of parameters, with complex and unknown dependence structures among variables. In analyzing such hypothesis testing procedures, gains in efficiency and power can be achieved by performing variable reduction on the set of hypotheses prior to testing. We present in this paper an approach using data-adaptive multiple testing that serves exactly this purpose. This approach applies data mining techniques to screen the full set of covariates on equally sized partitions of the whole sample via cross-validation. This generalized screening procedure is used to create average ranks for covariates, which are then used to generate a reduced (sub)set of hypotheses, from which we compute test statistics that are subsequently subjected to standard multiple testing corrections. The principal advantage of this methodology lies in its providing valid statistical inference without the \textit{a priori} specifying which hypotheses will be tested. Here, we present the theoretical details of this approach, confirm its validity via a simulation study, and exemplify its use by applying it to the analysis of data on microRNA differential expression.
BibTeX
@misc{cai2017data,
  author = {Weixin Cai and Nima S. Hejazi and Alan Hubbard},
  title = {{Data-adaptive statistics for multiple hypothesis testing in high-dimensional settings}},
  year = {2017},
  eprint = {1704.07008},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1704.07008},
}
Figure