Powerful t-tests in the presence of nonclassical measurement error

Dongwoo Kim, Daniel Wilhelm

Abstract

This paper proposes a powerful alternative to the t-test of the null hypothesis that a coefficient in linear regression is equal to zero when a regressor is mismeasured. We assume there are two contaminated measurements of the regressor of interest. We allow the two measurement errors to be nonclassical in the sense that they may both be correlated with the true regressor, they may be correlated with each other, and we do not require any location normalizations on the measurement errors. We propose a new maximal t-statistic that is formed from the regression of the outcome onto a maximally weighted linear combination of the two measurements. Critical values of the test are easily computed via a multiplier bootstrap. In simulations, we show that this new test can be significantly more powerful than t-statistics based on OLS or IV estimates. Finally, we apply the proposed test to a study of returns to education based on twins data from the UK. With our maximal t-test, we can discover statistically significant returns to education when standard t-tests do not.

Introduction

Measurement error (ME) is a pervasive challenge in the social and medical sciences. In many empirical settings, variables of interest—such as annual income or years of schooling—are self-reported, leading to data that contains "nonclassical" errors. Unlike random noise, these errors are often correlated with the true underlying value or with other measurements. For instance, high earners might systematically under-report their income.

Standard econometric tools, such as Ordinary Least Squares (OLS) or Instrumental Variables (IV), frequently lose significant statistical power in the presence of such complex errors. While researchers often have access to multiple measurements (e.g., a twin reporting their own education and their sibling's education), the choice of which measurement to use or how to combine them is typically arbitrary. This paper addresses this gap by introducing a robust, high-power testing framework designed specifically for scenarios where measurement errors are complex and dependent.

Methodology: The Maximal T-Statistic

The core innovation of this research is the maximal t-statistic. Rather than selecting a single measurement or relying on a simple average, the authors propose a data-driven approach to optimize the test's sensitivity.

The procedure involves creating a linear combination of two available measurements ($X$ and $Z$) using weights specifically chosen to maximize the resulting t-statistic. The paper demonstrates that under homoskedasticity (constant variance), a closed-form solution exists for these optimal weights. In more complex heteroskedastic cases, the problem is reduced to a manageable two-dimensional optimization.

Because the "maximum" of a set of statistics does not follow a standard t-distribution, the authors employ a multiplier bootstrap procedure. This method allows for the calculation of accurate critical values, ensuring the test maintains the correct size (false positive rate) while maximizing its power to detect real effects.

Experimental Evaluation

The proposed method was evaluated using both synthetic and real-world data:

  1. Monte Carlo Simulations: The authors generated 1,000 synthetic datasets ($n=200$) across seven scenarios. These scenarios varied the error variance and the degree of correlation between errors and true values.
  2. UK Twin Data: The study re-examined a dataset from the UK Adult Twin Registry. This dataset is a classic example of nonclassical measurement error, featuring self-reported and cross-reported education levels for monozygotic twins.

Key Insights and Results

The research yields several critical insights for empirical researchers:

  • Adaptive Weighting: The maximal approach is inherently adaptive. It does not require prior knowledge of which measurement is "cleaner"; the mathematical framework automatically assigns more weight to the more informative measurement.
  • Testing vs. Estimation: A significant theoretical finding is that while combining measurements may not narrow the "bounds" of the true coefficient $\beta$, it significantly enhances the ability to test whether $\beta$ differs from zero.
  • Superior Power: In simulations, the maximal t-test consistently outperformed OLS and IV tests. It was often the only method (aside from an "oracle" test with perfect information) to maintain high power as measurement error complexity increased.
  • Empirical Application: In the study of returns to schooling, previous research had suggested that the effect of education on wages was statistically insignificant after removing outliers. By applying the maximal t-test to the same data, the authors recovered statistically significant results. This suggests that previous "insignificant" findings were likely a consequence of low-power testing rather than a lack of an underlying effect.

Limitations and Future Directions

While powerful, the framework has specific boundaries. It currently focuses on linear combinations of measurements; while non-linear extensions are possible, they may increase sensitivity to outliers. Furthermore, the test requires that measurement errors are "exogenous" — meaning the errors themselves are not correlated with the final outcome's error term. Finally, the method is optimized for testing the null hypothesis of "no effect" ($\beta = 0$). Extending this to general hypotheses ($\beta = b$) remains a challenge because the parameter is not point-identified under these conditions.

Despite these limitations, the maximal t-test provides a significant advancement for researchers dealing with contaminated data, offering a way to "rescue" findings that might otherwise be lost to the noise of nonclassical measurement error.

Summary updated

Abstract

This paper proposes a powerful alternative to the t-test of the null hypothesis that a coefficient in linear regression is equal to zero when a regressor is mismeasured. We assume there are two contaminated measurements of the regressor of interest. We allow the two measurement errors to be nonclassical in the sense that they may both be correlated with the true regressor, they may be correlated with each other, and we do not require any location normalizations on the measurement errors. We propose a new maximal t-statistic that is formed from the regression of the outcome onto a maximally weighted linear combination of the two measurements. Critical values of the test are easily computed via a multiplier bootstrap. In simulations, we show that this new test can be significantly more powerful than t-statistics based on OLS or IV estimates. Finally, we apply the proposed test to a study of returns to education based on twins data from the UK. With our maximal t-test, we can discover statistically significant returns to education when standard t-tests do not.
BibTeX
@inproceedings{kim2024powerful,
  author = {Dongwoo Kim and Daniel Wilhelm},
  title = {{Powerful t-tests in the presence of nonclassical measurement error}},
  booktitle = {Econometric},
  year = {2024},
  doi = {10.1080/07474938.2024.2334166},
}
Figure