Nonparametric Tests for Treatment Effect Heterogeneity

Richard K. Crump, V. Joseph Hotz, Guido W. Imbens, Oscar A. Mitnik

Abstract

A large part of the recent literature on program evaluation has focused on estimation of the average effect of the treatment under assumptions of unconfoundedness or ignorability. However, researchers are often interested in the effects of programs beyond estimates of the overall average. It may be of substantive interest to investigate whether there is any subpopulation for which a program has a nonzero average effect, or whether there is heterogeneity in that effect. This paper develops two nonparametric tests: the first for the null hypothesis that the treatment has a zero average effect for any subpopulation defined by covariates, and the second for the null hypothesis that the average effect is identical for all subpopulations. Using sieve estimation, these tests are straightforward to implement and provide a robust framework for detecting varied program impacts.

Introduction

In the evaluation of social programs and clinical trials, the Average Treatment Effect (ATE) is the standard metric for success. However, relying solely on the ATE can be misleading. As noted in the seminal work of Rubin (1974), the impact of a treatment may vary significantly across different individuals. If a program benefits one group but harms another, the ATE might be zero, leading researchers to incorrectly conclude that the program is ineffective.

This paper addresses this gap by providing nonparametric tools to test for the presence of any effect and the presence of heterogeneous effects. These tests are particularly useful for assessing the "unconfoundedness" assumption—the idea that all relevant variables have been controlled for—by applying them to lagged outcomes where no effect should be present.

The Limitations of Average Effects

The core issue with the ATE is that it aggregates information, potentially washing out significant signals. Consider a scenario where a medical treatment saves 50% of patients but is fatal to the other 50%. The average effect is zero, yet the treatment is clearly doing something significant. This is the essence of treatment effect heterogeneity.

Furthermore, traditional parametric tests often rely on linear models. If the true relationship between covariates and outcomes is non-linear, these models may fail to detect effects simply because of misspecification. Nonparametric methods allow the data to "speak for itself" without being forced into a pre-defined functional form, avoiding the trap of failing to reject a null hypothesis simply because the researcher chose the wrong mathematical shape (like a straight line) to represent the effect.

Methodology: Sieve Estimation

The authors propose two specific tests:

  1. The Zero Conditional Average Treatment Effect (CATE) Test: This tests whether the treatment effect is zero for every subpopulation defined by observed characteristics ($x$).
  2. The Constant CATE Test: This tests whether the treatment effect is the same for everyone, regardless of their characteristics.

To implement these tests without assuming a specific functional form, the authors utilize sieve (or series) estimation. This technique approximates unknown regression functions using an expanding set of basis functions, such as power series. As the sample size grows, the number of terms ($K$) in the series increases, allowing for a more flexible fit.

While the underlying theory is complex, the resulting test statistics are quadratic forms. Under the null hypothesis, these statistics follow a standard chi-squared distribution in large samples. This is a significant advantage for practitioners, as it allows for implementation using standard econometric software without the need for complex custom algorithms.

Empirical Application: GAIN and WIN

The power of these tests is demonstrated using data from two landmark welfare-to-work experiments: the Greater Avenues for INdependence (GAIN) study and the Work INcentive (WIN) demonstrations. These datasets are benchmarks in labor economics because they involve random assignment, providing a clean environment to test new statistical methods.

GAIN Results

The GAIN study evaluated job training programs across four California counties. When applying traditional ATE tests, a significant effect was found only in Riverside County. However, when the authors applied their Zero Conditional test, they rejected the null hypothesis in three out of the four counties (Riverside, Alameda, and San Diego). This indicates that the programs in Alameda and San Diego were indeed having an impact, but those impacts were heterogeneous and thus invisible to simple averaging.

WIN Results

Similar patterns emerged in the WIN data. In several locations where traditional tests suggested the programs were ineffective, the nonparametric tests revealed significant treatment effects. This suggests that many historical programs labeled as "failures" based on ATE might have actually had meaningful, albeit varied, impacts on participants.

Discussion and Limitations

While these nonparametric tests offer a more nuanced view of program impacts, they come with specific requirements:

  • Selection of $K$: The performance of the test is sensitive to the number of terms ($K$) used in the series expansion. While the paper provides theoretical guidance, the optimal choice in finite samples remains a practical challenge.
  • Data Intensity: Nonparametric methods are inherently "data-hungry." To accurately estimate effects for various subpopulations, large datasets are required to avoid the "curse of dimensionality."
  • Assumption of Unconfoundedness: These tests still operate under the assumption that there are no unobserved variables influencing both treatment assignment and outcomes. However, they provide a rigorous way to perform "placebo tests" on lagged outcomes to check the plausibility of this assumption. If a test shows the program "affected" someone's earnings from years before the program started, the model's assumptions are likely broken.

Conclusion

By moving beyond the Average Treatment Effect, these nonparametric tests provide a more comprehensive toolkit for program evaluation. They allow researchers to detect signals that are often lost in the noise of aggregation and provide a robust check against model misspecification. For policymakers, this means a better understanding of not just if a program works, but for whom it might be working.

Summary updated

Abstract

A large part of the recent literature on program evaluation has focused on estimation of the\naverage effect of the treatment under assumptions of unconfoundedness or ignorability following the seminal work by Rubin (1974) and Rosenbaum and Rubin (1983). In many cases however, researchers are interested in the effects of programs beyond estimates of the overall average or the average for the subpopulation of treated individuals. It may be of\nsubstantive interest to investigate whether there is any subpopulation for which a program or treatment has a nonzero average effect, or whether there is heterogeneity in the effect of the treatment. The hypothesis that the average effect of the treatment is zero for all subpopulations is also important for researchers interested in assessing assumptions\nconcerning the selection mechanism. In this paper we develop two nonparametric tests. The\nfirst test is for the null hypothesis that the treatment has a zero average effect for any subpopulation defined by covariates. The second test is for the null hypothesis that the average effect conditional on the covariates is identical for all subpopulations, in other words, that there is no heterogeneity in average treatment effects by covariates. Sacrificing some\ngenerality by focusing on these two specific null hypotheses we derive tests that are\nstraightforward to implement.
BibTeX
@misc{crump2006nonparametric,
  author = {Richard K. Crump and V. Joseph Hotz and Guido W. Imbens and Oscar A. Mitnik},
  title = {{Nonparametric Tests for Treatment Effect Heterogeneity}},
  year = {2006},
  doi = {10.3386/t0324},
}
Figure