Abstract
A large part of the recent literature on program evaluation has focused on estimation of the average effect of the treatment under assumptions of unconfoundedness or ignorability. However, researchers are often interested in the effects of programs beyond estimates of the overall average. It may be of substantive interest to investigate whether there is any subpopulation for which a program has a nonzero average effect, or whether there is heterogeneity in that effect. This paper develops two nonparametric tests: the first for the null hypothesis that the treatment has a zero average effect for any subpopulation defined by covariates, and the second for the null hypothesis that the average effect is identical for all subpopulations. Using sieve estimation, these tests are straightforward to implement and provide a robust framework for detecting varied program impacts.
Introduction
In the evaluation of social programs and clinical trials, the Average Treatment Effect (ATE) is the standard metric for success. However, relying solely on the ATE can be misleading. As noted in the seminal work of Rubin (1974), the impact of a treatment may vary significantly across different individuals. If a program benefits one group but harms another, the ATE might be zero, leading researchers to incorrectly conclude that the program is ineffective.
This paper addresses this gap by providing nonparametric tools to test for the presence of any effect and the presence of heterogeneous effects. These tests are particularly useful for assessing the "unconfoundedness" assumption—the idea that all relevant variables have been controlled for—by applying them to lagged outcomes where no effect should be present.
The Limitations of Average Effects
The core issue with the ATE is that it aggregates information, potentially washing out significant signals. Consider a scenario where a medical treatment saves 50% of patients but is fatal to the other 50%. The average effect is zero, yet the treatment is clearly doing something significant. This is the essence of treatment effect heterogeneity.
Furthermore, traditional parametric tests often rely on linear models. If the true relationship between covariates and outcomes is non-linear, these models may fail to detect effects simply because of misspecification. Nonparametric methods allow the data to "speak for itself" without being forced into a pre-defined functional form, avoiding the trap of failing to reject a null hypothesis simply because the researcher chose the wrong mathematical shape (like a straight line) to represent the effect.
Methodology: Sieve Estimation
The authors propose two specific tests:
- The Zero Conditional Average Treatment Effect (CATE) Test: This tests whether the treatment effect is zero for every subpopulation defined by observed characteristics ($x$).
- The Constant CATE Test: This tests whether the treatment effect is the same for everyone, regardless of their characteristics.
To implement these tests without assuming a specific functional form, the authors utilize sieve (or series) estimation. This technique approximates unknown regression functions using an expanding set of basis functions, such as power series. As the sample size grows, the number of terms ($K$) in the series increases, allowing for a more flexible fit.
While the underlying theory is complex, the resulting test statistics are quadratic forms. Under the null hypothesis, these statistics follow a standard chi-squared distribution in large samples. This is a significant advantage for practitioners, as it allows for implementation using standard econometric software without the need for complex custom algorithms.
Empirical Application: GAIN and WIN
The power of these tests is demonstrated using data from two landmark welfare-to-work experiments: the Greater Avenues for INdependence (GAIN) study and the Work INcentive (WIN) demonstrations. These datasets are benchmarks in labor economics because they involve random assignment, providing a clean environment to test new statistical methods.
GAIN Results
The GAIN study evaluated job training programs across four California counties. When applying traditional ATE tests, a significant effect was found only in Riverside County. However, when the authors applied their Zero Conditional test, they rejected the null hypothesis in three out of the four counties (Riverside, Alameda, and San Diego). This indicates that the programs in Alameda and San Diego were indeed having an impact, but those impacts were heterogeneous and thus invisible to simple averaging.
WIN Results
Similar patterns emerged in the WIN data. In several locations where traditional tests suggested the programs were ineffective, the nonparametric tests revealed significant treatment effects. This suggests that many historical programs labeled as "failures" based on ATE might have actually had meaningful, albeit varied, impacts on participants.
Discussion and Limitations
While these nonparametric tests offer a more nuanced view of program impacts, they come with specific requirements:
- Selection of $K$: The performance of the test is sensitive to the number of terms ($K$) used in the series expansion. While the paper provides theoretical guidance, the optimal choice in finite samples remains a practical challenge.
- Data Intensity: Nonparametric methods are inherently "data-hungry." To accurately estimate effects for various subpopulations, large datasets are required to avoid the "curse of dimensionality."
- Assumption of Unconfoundedness: These tests still operate under the assumption that there are no unobserved variables influencing both treatment assignment and outcomes. However, they provide a rigorous way to perform "placebo tests" on lagged outcomes to check the plausibility of this assumption. If a test shows the program "affected" someone's earnings from years before the program started, the model's assumptions are likely broken.
Conclusion
By moving beyond the Average Treatment Effect, these nonparametric tests provide a more comprehensive toolkit for program evaluation. They allow researchers to detect signals that are often lost in the noise of aggregation and provide a robust check against model misspecification. For policymakers, this means a better understanding of not just if a program works, but for whom it might be working.
Summary updated