Matching on the Estimated Propensity Score

Alberto Abadie, Guido W. ImbensEconometrica 2016 · DOI

TL;DR

This paper tackles a crucial issue in causal inference: how to accurately estimate treatment effects when the propensity score (the probability of receiving treatment) must first be estimated from data. The authors derive the correct large-sample distribution for propensity score matching estimators, showing that ignoring the estimation of the propensity score leads to incorrect variance estimates. They provide an adjustment that improves the efficiency and accuracy of these estimates, particularly for the average treatment effect (ATE).

What Problem Does This Paper Address?

In observational studies, researchers often want to understand the causal effect of a treatment or intervention. A common method to approximate this is propensity score matching, introduced by Rosenbaum and Rubin (1983). The core idea is to match individuals who received the treatment with similar individuals who did not, where "similarity" is defined by their propensity score – the probability of receiving treatment given their observed characteristics. This helps to control for confounding variables.

However, in practice, the true propensity score is rarely known. It must be estimated from the data, typically using a regression model (like logit or probit). This two-step process – estimating the propensity score first, then using it for matching – introduces an additional layer of uncertainty. The standard statistical methods for analyzing matching estimators assume the propensity score is known, which can lead to incorrect inferences (e.g., confidence intervals that are too wide or too narrow) when the propensity score is estimated. This paper addresses this gap by deriving the correct large-sample distribution and providing an adjustment for the variance of propensity score matching estimators when the propensity score is estimated.

Methods

The authors employ advanced econometric and statistical techniques to derive the large-sample distribution of propensity score matching estimators. Key methods include:

  • Martingale Representations: They use martingale representations of matching estimators, which are powerful tools for analyzing the asymptotic behavior of statistics in time-series or sequential data.
  • Asymptotic Analysis: The core of the paper involves deriving the limiting distribution of the estimators under large sample sizes. This requires careful handling of the two-step estimation procedure.
  • Decomposition of Estimators: The estimators are decomposed into components that allow for separate analysis of the matching part and the propensity score estimation part.
  • Adjustment for Estimation Error: A novel adjustment to the asymptotic variance is derived to account for the uncertainty introduced by estimating the propensity score. This adjustment is shown to improve efficiency for the ATE.
  • Simulation Study: A Monte Carlo simulation is used to empirically validate the theoretical results and compare the performance of the proposed adjusted estimators against unadjusted ones.

Datasets

Not discussed in the paper. The paper focuses on theoretical derivations and uses a simulated dataset for its empirical illustration.

Key Insights and Ideas

  • The "Curse" of Estimated Propensity Scores: Estimating the propensity score before matching is not benign. It systematically affects the asymptotic distribution of matching estimators, and ignoring this effect leads to incorrect standard errors.
  • Efficiency Gains for ATE: For estimating the Average Treatment Effect (ATE), matching on the estimated propensity score is actually more efficient in large samples than matching on the true propensity score. This is because the estimation process can sometimes reduce the variance of the matching estimator.
  • Complexity for ATET: The situation is more nuanced for estimating the Average Treatment Effect on the Treated (ATET). The adjustment for estimation error can be positive or negative, meaning that ignoring the estimation error can lead to confidence intervals that are either too wide or too narrow, depending on the data.
  • Bootstrap Failure: The paper references prior work (Abadie and Imbens, 2008) showing that the standard bootstrap is not generally valid for propensity score matching estimators, highlighting the need for analytical solutions like the one presented here.

Key Results

  • The paper derives the correct large-sample distribution for propensity score matching estimators when the propensity score is estimated.
  • An adjustment to the asymptotic variance is proposed. For the ATE estimator, this adjustment is negative, leading to a more efficient estimator.
  • The simulation results confirm that the proposed adjusted variance estimator leads to confidence intervals with coverage rates close to nominal levels, whereas unadjusted methods can be biased and have coverage rates that are too high. Specifically, the adjusted estimator adj.N(ON) in the simulation showed unbiasedness and correct coverage, while the unadjusted variance estimator σ²(θ) was biased and too large.

Limitations and Future Work

  • Assumptions: The derivations rely on standard regularity conditions for parametric models and large-sample approximations. The applicability of the results might be limited in small samples or when these assumptions are severely violated.
  • Generalizability of Adjustments: While the paper provides an adjustment for the ATE, the sign and impact of the adjustment for other treatment effect parameters (like ATET) are data-dependent and require careful consideration.
  • Future Work: The authors suggest that their results can be extended to other matching estimators and treatment parameters. The paper also highlights the need for robust inference methods when propensity scores are estimated.

Summary updated

Abstract

Propensity score matching estimators (Rosenbaum and Rubin (1983)) are widely used in evaluation research to estimate average treatment effects. In this article, we derive the large sample distribution of propensity score matching estimators. Our derivations take into account that the propensity score is itself estimated in a first step, prior to matching. We prove that first step estimation of the propensity score affects the large sample distribution of propensity score matching estimators, and derive adjustments to the large sample variances of propensity score matching estimators of the average treatment effect (ATE) and the average treatment effect on the treated (ATET). The adjustment for the ATE estimator is negative (or zero in some special cases), implying that matching on the estimated propensity score is more efficient than matching on the true propensity score in large samples. However, for the ATET estimator, the sign of the adjustment term depends on the data generating process, and ignoring the estimation error in the propensity score may lead to confidence intervals that are either too large or too small.
BibTeX
@inproceedings{abadie2016matching,
  author = {Alberto Abadie and Guido W. Imbens},
  title = {{Matching on the Estimated Propensity Score}},
  booktitle = {Econometrica},
  year = {2016},
  doi = {10.3982/ecta11293},
}

Loading related content...

Figure