Abstract
We propose a novel technique to boost the power of testing a high-dimensional vector $H_0: \boldsymbol{\theta}=0$ against sparse alternatives where the null hypothesis is violated by only a few components. Existing tests based on quadratic forms, such as the Wald statistic, often suffer from low power due to the accumulation of estimation errors across high-dimensional parameters. Conversely, tests designed for sparse alternatives, such as thresholding and extreme-value tests, often suffer from size distortions due to slow convergence. We introduce a "power enhancement component" that is zero under the null hypothesis with high probability but diverges quickly under sparse alternatives. This component is combined with an asymptotically pivotal statistic to strengthen power while maintaining a well-behaved null distribution.
Introduction
In the era of high-dimensional econometrics, researchers frequently test hypotheses involving thousands of variables. A common example is testing the "alpha" (excess return) of every stock in a major index like the S&P 500. In such cases, the number of parameters ($N$) often exceeds the number of time observations ($T$).
Traditional statistical tests face a significant hurdle in these settings: they lose power as they accumulate noise from every estimated component. If a signal is "sparse"—meaning only a few variables out of hundreds actually violate the null—the collective noise of the null components can drown out the signal entirely. This paper introduces a "Power Enhancement" (PE) framework to address this sensitivity gap without increasing false positives.
The Problem with High-Dimensional Testing
When testing $H_0: \boldsymbol{\theta} = 0$, the standard approach utilizes a quadratic form like the Wald statistic. However, in high dimensions, the estimation errors from $N$ components accumulate. If the true signal is concentrated in only 2 or 3 variables out of 500, the Wald statistic's power is severely diluted.
While existing "sparse" tests (like thresholding) attempt to focus on outliers, they often require stringent regularity conditions or computationally intensive bootstraps to derive null distributions. In finite samples, these methods frequently exhibit "size distortion," rejecting the null hypothesis far more often than the nominal level suggests.
Methodology: The Power Enhancement Component
The proposed solution is a hybrid statistic: $J = J_0 + J_1$.
- $J_1$ (The Base): This is an asymptotically pivotal test statistic, such as a standardized Wald test. It ensures the test has a known, stable distribution under the null hypothesis, allowing for standard critical values.
- $J_0$ (The Booster): This is the "Power Enhancement Component," constructed using a screening technique with a "high-criticism" threshold.
The construction of $J_0$ is the primary innovation. The threshold is calibrated so that, if the null hypothesis is true, the probability of any component exceeding the threshold is asymptotically zero. Consequently, $J_0$ remains zero under the null with high probability. However, if even a few components of $\boldsymbol{\theta}$ are significantly non-zero, $J_0$ diverges rapidly, dominating the total statistic and ensuring a rejection of the null.
This framework allows researchers to enjoy the massive power boost of screening-based methods while relying on the simple, standard null distributions of pivotal statistics.
Empirical Evidence and Results
The authors validated the PE method through both Monte Carlo simulations and an empirical study of financial markets.
Simulation Results
Using models based on the Fama-French three-factor framework, the simulations compared the PE test against standard Wald-type tests. In scenarios with sparse alternatives, the PE test achieved nearly 100% power (detection rate), whereas the standard Wald test's power hovered between 30% and 60%. This demonstrates the PE test's ability to "find the needle in the haystack" that traditional methods miss.
Financial Application: S&P 500 Alphas
The method was applied to monthly excess returns of S&P 500 constituents from 1980 to 2012 to test Carhart’s four-factor model. The PE test rejected the null hypothesis (that all alphas are zero) significantly more frequently than standard tests. Interestingly, the screening component identified that, on average, only about 5.2 stocks at any given time were responsible for the violation of the null. This confirms that market mispricing is a sparse phenomenon, making the PE test an ideal diagnostic tool.
Conclusion
The Power Enhancement framework provides a robust solution for high-dimensional testing. By augmenting traditional pivotal statistics with a screening-based booster, it achieves high power against sparse alternatives while maintaining strict size control. As a byproduct, the method identifies the specific components violating the null, providing researchers with immediate diagnostic insights into which variables are driving the results.
Summary updated