TL;DR
This paper introduces a new Difference-in-Differences (DID) estimator that leverages the Covariate Balancing Propensity Score (CBPS). Unlike traditional methods, this estimator provides "double robustness" for both the treatment effect estimate and the statistical inference (confidence intervals), while outperforming standard models when both the propensity score and outcome models are slightly misspecified.
What Problem Does This Paper Address?
The Difference-in-Differences (DID) design is a staple in econometrics for estimating the Average Treatment Effect on the Treated (ATT). Its validity hinges on the Parallel Trends Assumption (PTA)—the idea that, without treatment, the average outcomes of the treated and control groups would have evolved identically over time.
In the real world, groups often differ in ways that make this assumption implausible unless we control for pre-treatment covariates (e.g., age, education). Researchers typically use one of three approaches to handle these covariates:
- Outcome Regression (OR): Modeling how outcomes change.
- Inverse Probability Weighting (IPW): Modeling the probability of receiving treatment.
- Augmented IPW (AIPW): Combining both for "double robustness" (the estimate is consistent if at least one model is correct).
However, AIPW has a "hidden" weakness: if one model is wrong, the standard methods for calculating confidence intervals can become invalid. Furthermore, if both models are even slightly "off" (local misspecification), the performance of AIPW can degrade significantly. This paper seeks a more stable, robust alternative.
Methods
The authors propose the CBPS DID estimator. The core innovation lies in how the propensity score (the probability of being treated) is estimated.
- Covariate Balancing: Instead of just trying to predict who got treated (like standard maximum likelihood estimation), CBPS solves for weights that force the covariates to be balanced between the treatment and comparison groups.
- Double Robustness in Inference: The authors prove that the "influence function" (the math used to calculate variance) for their estimator remains invariant even if one of the working models is misspecified. This allows for reliable hypothesis testing where AIPW might fail.
- Handling Local Misspecification: Using a Taylor expansion framework, they demonstrate that when both the outcome and propensity models are slightly wrong, the CBPS DID estimator converges to the true value at a faster rate than the AIPW DID estimator.
Datasets
The paper evaluates the estimator using two primary methods:
- Monte Carlo Simulations: Five distinct scenarios (DGPs) ranging from perfectly specified models to cases where both models are locally or globally misspecified.
- Empirical Application: The classic LaLonde (1986) labor training study. They use the National Supported Work (NSW) program data as a "pseudo-treatment" group and combine it with:
- Current Population Survey (CPS) data for a comparison group.
- Panel Study of Income Dynamics (PSID) data for an alternative comparison group.
- Note: Because they use a setup where the "true" effect should be zero, they can measure exactly how much bias each estimator introduces.
Key Insights and Ideas
- Balance Over Prediction: The "aha moment" is that for causal inference, having groups that look identical (balance) is often more important than having a model that perfectly predicts treatment assignment.
- The "Arbitrary Gamma" Advantage: In the math of the CBPS estimator, a specific nuisance parameter ($\gamma$) can effectively take any value. The authors show that by choosing a value that satisfies certain balancing conditions, they can "cancel out" the error terms that usually arise from model misspecification.
- Invariance is Key: Most researchers focus on the point estimate. This paper highlights that inference robustness—the ability to trust your p-values when your model isn't perfect—is a major theoretical advantage of the CBPS approach over AIPW.
Key Results
- Superior Coverage: In simulations where the outcome model was misspecified (DGP3), the CBPS estimator maintained a 94.7% coverage probability for its 95% confidence interval, while the standard Outcome Regression (OR) dropped to 82.6%.
- Reduced Bias: In the empirical study using the Dehejia-Wahba (DW) sample, the CBPS DID estimator was remarkably accurate, showing a deviation of only 2 (with a standard error of 519) from the true ATT of zero, whereas the AIPW estimator showed a deviation of 69.
- Efficiency: Across almost all tests, CBPS produced smaller standard errors than the IPW approach, proving it is a more "efficient" use of the available data.
Limitations and Future Work
- High-Dimensional Data: The current paper focuses on settings with a manageable number of covariates. The authors acknowledge that in "Big Data" settings with hundreds or thousands of variables, traditional CBPS might struggle.
- Machine Learning Integration: The authors suggest that an "intriguing extension" would be adapting this CBPS DID framework to high-dimensional settings using Double Machine Learning (DML) to handle the first-step estimations. This is noted as an area of ongoing research.
Summary updated