Why High-Order Polynomials Should Not Be Used in Regression Discontinuity Designs

Andrew Gelman, Guido W. ImbensJournal of Business and Economic Statistics 2017 · DOI

TL;DR

This paper argues against using high-order global polynomials in regression discontinuity designs (RDD), a popular method for estimating causal effects. The authors demonstrate that these methods lead to unreliable estimates due to noisy weights, extreme sensitivity to the chosen polynomial degree, and poor confidence interval coverage, recommending local linear or quadratic approaches instead.

What Problem Does This Paper Address?

Regression discontinuity designs (RDD) are a powerful tool for causal inference when treatment assignment is determined by a sharp cutoff on a continuous variable (the "forcing variable"). Researchers often use polynomial regressions to model the relationship between the forcing variable and the outcome, aiming to estimate the "jump" or discontinuity at the cutoff, which represents the causal effect. A common practice is to fit high-order global polynomials (e.g., cubic, quartic, or higher) to the entire dataset on either side of the cutoff. However, the authors observe that this approach can lead to misleading results and argue that there's a lack of clear theoretical justification for choosing the specific order of these high-degree polynomials. The core problem is that these high-order global polynomials, while seemingly sophisticated, can produce problematic estimates and inferences that don't accurately reflect the underlying causal effect.

Methods

The paper's arguments are primarily theoretical and empirical, focusing on the properties of different RDD estimation methods. The authors' approach involves:

  • Weight Function Analysis: They analyze the "weights" assigned to observations by different estimators. In RDD, estimators can be viewed as weighted averages of outcomes. The authors show that high-order global polynomials can assign very large, and often oscillating, weights to observations far from the discontinuity, which is problematic.
  • Sensitivity Analysis: They demonstrate how estimates change drastically when the degree of the global polynomial is altered, highlighting a lack of robustness.
  • Simulation and Empirical Examples: The authors use real-world datasets and simulated scenarios to illustrate their points. They compare the performance of global polynomial estimators (of varying degrees) against local linear and local quadratic estimators.
  • Confidence Interval Coverage: They conduct simulations to assess whether confidence intervals generated by different methods achieve their nominal coverage rates (e.g., a 95% confidence interval should contain the true effect 95% of the time).

The paper advocates for local polynomial regression (linear or quadratic) as a superior alternative, emphasizing the importance of choosing a bandwidth that focuses on observations close to the threshold.

Datasets

The paper uses the following datasets:

  • Matsudaira Data: This dataset, previously analyzed by Matsudaira (2008), concerns the effect of a mandatory summer program on subsequent academic achievement. It's used to illustrate the noisy weights and sensitivity to polynomial order.
  • Jacob-Lefgren Data: This dataset, previously analyzed by Jacob and Lefgren (2004), also relates to the effect of a summer school program on student achievement. It's used to further demonstrate the variability in estimates from global polynomials.
  • Lalonde Data: This dataset, previously used by Lalonde (1986) and Dehejia and Wahba (1999), contains information on yearly earnings. The authors use it to construct a scenario without an actual discontinuity to test the inference properties (confidence interval coverage) of different RDD estimators.

No new benchmarks are introduced; the focus is on re-evaluating existing methods using established datasets.

Key Insights and Ideas

  • Noisy Weights are a Red Flag: The most striking insight is how high-order global polynomials can assign extreme weights to data points far from the discontinuity. This is counterintuitive because these distant points should have little bearing on the estimated effect at the cutoff. These noisy weights directly lead to unstable and unreliable estimates.
  • Sensitivity to Polynomial Degree is a Sign of Trouble: The fact that estimates change dramatically with small changes in the polynomial order (e.g., from degree 3 to 4) suggests that the model is overfitting or capturing spurious patterns rather than the true underlying relationship. This makes it difficult to justify any particular choice of polynomial degree.
  • Confidence Intervals Can Be Deceiving: The simulation results showing that global polynomial estimators frequently produce confidence intervals that fail to capture the true effect (over-rejection in hypothesis tests) are a critical finding. This means researchers using these methods might incorrectly conclude there's a significant effect when there isn't, or vice-versa.
  • Local is Generally Better: The paper strongly advocates for local polynomial estimators (linear or quadratic) with an appropriate bandwidth. These methods focus on data closest to the threshold, naturally assigning lower weights to distant observations, leading to more stable estimates and reliable inference.

Key Results

  • Weight Visualization: Figures 1a and 1b clearly show how global polynomial weights oscillate and can be very large for distant observations, while local linear weights are well-behaved and concentrated near the threshold.
  • Estimate Variability: Tables 2 and 3 demonstrate that estimates from global polynomials of different degrees can vary significantly, often by more than their standard errors, indicating poor precision and reliability. Local polynomial estimates are generally more stable.
  • Confidence Interval Performance: Table 6 shows that for a scenario with no true discontinuity, global polynomial estimators lead to rejection rates far exceeding the nominal 5% level (e.g., 49% for degree 1, down to 11% for degrees 5 and 6, but still much higher than 5%). Local linear and quadratic estimators achieve rejection rates close to the nominal 5% (around 7%).

Limitations and Future Work

  • Limitations: The authors acknowledge that their arguments are based on the properties of estimators and their empirical performance. While they provide strong evidence, the theoretical underpinnings of why high-order polynomials behave this way are implicitly discussed through the weight function analysis. The paper doesn't delve into specific theoretical proofs for all cases but relies on illustrative examples and simulations.
  • Future Work: The paper implicitly suggests that researchers should routinely present the implicit weights from their RDD estimates to allow for scrutiny. It also strongly recommends the use of local linear or quadratic polynomial estimators, or other smooth functions, as preferred methods for RDD analysis. The authors' recommendation is to move away from global high-order polynomials entirely.

Summary updated

Abstract

It is common in regression discontinuity analysis to control for third, fourth, or higher-degree polynomials of the forcing variable. There appears to be a perception that such methods are theoretically justified, even though they can lead to evidently nonsensical results. We argue that controlling for global high-order polynomials in regression discontinuity analysis is a flawed approach with three major problems: it leads to noisy estimates, sensitivity to the degree of the polynomial, and poor coverage of confidence intervals. We recommend researchers instead use estimators based on local linear or quadratic polynomials or other smooth functions.
BibTeX
@article{gelman2017why,
  author = {Andrew Gelman and Guido W. Imbens},
  title = {{Why High-Order Polynomials Should Not Be Used in Regression Discontinuity Designs}},
  journal = {Journal of Business and Economic Statistics},
  year = {2017},
  doi = {10.1080/07350015.2017.1366909},
}
Figure