Quantile treatment effects in difference in differences models under dependence restrictions and with only two time periods

Brantly Callaway, Tong Li, Tatsushi Oka

TL;DR

This paper introduces a novel framework for identifying Conditional Quantile Treatment Effects on the Treated (CQTT) using only two periods of data. By introducing a "Copula Invariance" assumption, the authors allow researchers to move beyond average effects and see how a policy impacts the entire distribution of outcomes—such as whether a minimum wage hike helps or hurts the lowest earners—even when experimental data is unavailable.

What Problem Does This Paper Address?

In policy evaluation, the "gold standard" is often the Difference-in-Differences (DiD) approach, which typically identifies the Average Treatment Effect on the Treated (ATT). However, averages can be misleading. A policy might have a zero average effect but significantly help the poor while hurting the middle class.

To understand these distributional impacts, researchers look for Quantile Treatment Effects (QTE). The challenge is that identifying what the distribution of the treated group would have been without the treatment (the counterfactual) is much harder than identifying a counterfactual average. Most existing methods for QTE require either experimental data, which is rare, or at least three periods of panel data to model how individuals' outcomes evolve over time. This paper tackles the common "data-poor" scenario where a researcher has only two periods: one before the treatment and one after.

Methods

The authors propose a two-step identification strategy based on two primary assumptions:

  1. Distributional Difference in Differences: This extends the standard "parallel trends" assumption. Instead of just assuming the average change is the same for both groups, it assumes the entire distribution of the change in untreated potential outcomes is the same for both the treated and untreated groups (conditional on covariates).
  2. Copula Invariance (The Key Innovation): Identifying the counterfactual distribution requires knowing the dependence (the "copula") between an individual's starting outcome and their subsequent change. Since we can't observe this for the treated group, the authors assume this dependence structure is the same as the one observed in the untreated group. Essentially, if the "best" performers in the control group saw the biggest gains, we assume the "best" performers in the treated group would have also seen the biggest gains in the absence of treatment.

For estimation, the authors use conditional empirical distributions. To handle statistical noise and provide confidence intervals, they propose a uniform inference procedure based on the exchangeable bootstrap, which is shown to be valid when covariates (like race or education) are discrete.

Datasets

  • Monte Carlo Simulations: The authors use synthetic Data Generating Processes (DGPs) to compare their model against the "Changes in Changes" (CIC) model, a popular alternative.
  • Current Population Survey (CPS): The empirical application uses individual-level panel data from the CPS Merged Outgoing Rotation Groups (1999–2007). They specifically focus on the 2006–2007 period to analyze the impact of state-level minimum wage increases.

Key Insights and Ideas

  • Borrowing Ranks: The "aha moment" is the use of the copula. It allows the researcher to "borrow" the rank-dependency from the control group. This acknowledges that people at different parts of the initial distribution (e.g., low vs. high earners) might experience different growth paths.
  • The Power of Two: Most panel data methods are "data hungry." Proving that you can get point-identification with just two periods makes this tool much more accessible for real-world policy analysis where long-running panels are rare.
  • Conditioning Matters: By making the model conditional on covariates, the "parallel trends" assumption becomes much more plausible. For example, it’s more likely that the earnings of two people with the same education level will follow a similar path than two people chosen at random.

Key Results

  • Simulation Performance: In simulations, the proposed estimator (DDID) was less biased than the CIC model, particularly at the tails of the distribution (the 0.9th quantile).
  • Minimum Wage Application: The authors analyzed the effect of 2007 minimum wage hikes across eight subgroups (defined by race, gender, and education).
  • They could only reject the "no effect" null hypothesis for 3 of the 8 groups: white females with college degrees, non-white males without college degrees, and non-white females without college degrees.
  • The Surprise: For these groups, the effect on earnings was actually negative at the lower end of the distribution. While the minimum wage increases the hourly rate, the authors suggest it may have led to a significant enough decrease in hours worked that total earnings actually fell for the lowest-earning members of these subgroups.

Limitations and Future Work

  • Discrete Covariates: The current inference results are optimized for discrete covariates. While many economic variables (race, sex, education level) are discrete, extending this to continuous covariates without hitting the "curse of dimensionality" remains a challenge.
  • Testability: Like all DiD models, the core assumptions (Distributional DiD and Copula Invariance) are not directly testable in a two-period setup. The authors suggest using "placebo tests" on earlier periods if more data becomes available.
  • Wage vs. Hours: The empirical results highlight that "earnings" is a composite of wages and hours. Future work could apply this method to wages and hours separately to disentangle the labor supply and demand effects more clearly.

Summary updated

Abstract

This paper shows that the Conditional Quantile Treatment Effect on the Treated can be identified using a combination of (i) a conditional Distributional Difference in Differences assumption and (ii) an assumption on the conditional dependence between the change in untreated potential outcomes and the initial level of untreated potential outcomes for the treated group. The second assumption recovers the unknown dependence from the observed dependence for the untreated group. We also consider estimation and inference in the case where all of the covariates are discrete. We propose a uniform inference procedure based on the exchangeable bootstrap and show its validity. We conclude the paper by estimating the effect of state-level changes in the minimum wage on the distribution of earnings for subgroups defined by race, gender, and education.
BibTeX
@article{callaway2018quantile,
  author = {Brantly Callaway and Tong Li and Tatsushi Oka},
  title = {{Quantile treatment effects in difference in differences models under dependence restrictions and with only two time periods}},
  journal = {Journal of Econometrics},
  year = {2018},
  doi = {10.1016/j.jeconom.2018.06.008},
}

Loading related content...

Figure