Abstract
We consider two alternative tests to the Higher Criticism (HC) test of Donoho and Jin (2004) for high-dimensional means under the sparsity of the nonzero means for sub-Gaussian distributed data with unknown column-wise dependence. The two alternative test statistics are constructed by first thresholding $L_{1}$ and $L_{2}$ statistics based on the sample means, respectively, followed by maximizing over a range of thresholding levels to make the tests adaptive to the unknown signal strength and sparsity. The two alternative tests can attain the same detection boundary of the Higher Criticism test which was established for uncorrelated Gaussian data. It is demonstrated that the maximal $L_{2}$-thresholding test is at least as powerful as the maximal $L_{1}$-thresholding test, and both the maximal $L_{2}$ and $L_{1}$-thresholding tests are at least as powerful as the Higher Criticism test.
Introduction
In the era of high-throughput data collection, researchers in fields such as genomics, finance, and signal processing frequently encounter the "large $p$, small $n$" problem. Here, the number of variables ($p$) vastly exceeds the number of observations ($n$). A fundamental challenge in this context is the detection of sparse and faint signals—situations where only a tiny fraction of the variables have non-zero means, and those means are only slightly larger than the surrounding noise.
The "Higher Criticism" (HC) test, pioneered by Donoho and Jin (2004), has long been the standard for this "needle in a haystack" problem. However, the original HC test relies on two restrictive assumptions: that the data follows a Gaussian distribution and that the variables are independent. In real-world applications, variables often exhibit complex dependencies, such as genes operating within shared biological pathways. This paper proposes two alternative tests based on $L_1$ and $L_2$ thresholding that extend the detection capabilities of HC to sub-Gaussian data with unknown column-wise dependence.
Methodology
Thresholding and Statistics
The authors propose two new test statistics, $\hat{\mathcal{M}}_{1n}$ and $\hat{\mathcal{M}}_{2n}$, which utilize $L_1$ and $L_2$ norms respectively. The core mechanism involves a thresholding procedure that filters out variables whose sample means do not exceed a specific level. This effectively ignores the "pure noise" variables that would otherwise dilute the signal.
- $L_1$ Statistic: This test sums the absolute values of the exceedances (the values remaining after thresholding).
- $L_2$ Statistic: This test sums the squares of the exceedances.
Adaptivity through Maximization
A significant hurdle in thresholding is that the optimal threshold depends on the unknown signal strength ($r$) and the degree of sparsity ($\beta$). To overcome this, the authors maximize the standardized statistics over a wide range of potential thresholding levels. This maximization ensures the tests are adaptive, allowing them to remain effective across various signal-to-noise landscapes without requiring prior knowledge of the signal's characteristics.
Robustness to Dependence
The research specifically addresses "weak dependence," modeled through $\rho$-mixing sequences. The authors provide a theoretical breakthrough by demonstrating that as the threshold increases, the events where variables exceed that threshold become increasingly independent. This "independence in the extremes" allows the $L_1$ and $L_2$ tests to maintain their validity and size (false positive rate) even when the underlying data is correlated.
Key Insights and Results
The Power Hierarchy
The most critical finding of the study is the establishment of a clear hierarchy in statistical power:
Maximal $L_2$ Thresholding $\ge$ Maximal $L_1$ Thresholding $\ge$ Higher Criticism.
While all three tests are theoretically capable of reaching the "optimal detection boundary"—the mathematical limit of detectability in high-dimensional space—the $L_2$ test is significantly more sensitive once the signal crosses that boundary. By squaring the exceedances, the $L_2$ test assigns greater weight to larger, more likely signals, thereby boosting the signal-to-noise ratio more effectively than the HC test, which primarily functions as a counting mechanism.
Numerical Simulations
To validate these theoretical claims, the authors conducted extensive simulations with variables up to $p = 20,000$ and sample sizes as small as $n = 30$. They varied parameters for sparsity, signal strength, and correlation (using $\rho = 0.3$ and $0.5$). In every scenario, the $L_2$ test outperformed the others, particularly in cases of extreme sparsity where the HC test often failed to detect any signal.
Conclusion
This paper provides a robust alternative to the Higher Criticism test for modern, high-dimensional datasets. By employing $L_1$ and $L_2$ thresholding and maximizing over threshold levels, the proposed tests offer superior power and adaptivity. Crucially, their ability to handle sub-Gaussian distributions and unknown dependencies makes them highly applicable to practical data analysis where the strict assumptions of independence and normality rarely hold. Future research may further refine these methods for cases of "strong" or long-range dependence and truly heavy-tailed distributions.
Summary updated