Flexible Sensitivity Analysis for Observational Studies Without Observable Implications

Alexander Franks, Alexander D’Amour, Avi Feller

TL;DR

This paper introduces a new framework for sensitivity analysis in observational causal inference that cleanly separates model checking from sensitivity analysis. By extending Tukey's factorization, it allows for flexible modeling of observed data while providing interpretable sensitivity parameters, making it easier to assess the robustness of causal conclusions to unmeasured confounding.

What Problem Does This Paper Address?

In observational studies, we often want to understand the causal effect of a treatment or exposure on an outcome. A major hurdle is "unconfoundedness," the assumption that all common causes of the treatment and outcome have been measured and controlled for. This assumption is untestable from the data alone. Sensitivity analysis is crucial for assessing how much our conclusions would change if this assumption were violated. However, many existing sensitivity analysis methods are either difficult to implement with modern flexible statistical models or they blur the line between checking the fit of the observed data model and assessing sensitivity to unmeasured confounding. This paper addresses this tension by proposing a framework that allows for flexible observed data models and a clear separation of what can be identified from the data versus what must be assumed.

Methods

The core of the paper's contribution is the application of Tukey's factorization to causal inference. This statistical technique, originally developed for missing data problems, allows for the joint distribution of potential outcomes to be decomposed into parts that are identifiable from the data and parts that are not.

Here's a breakdown of their approach:

  • Tukey's Factorization for Causal Inference: They extend Tukey's factorization to the causal setting, which involves potential outcomes for both treatment and control groups. This factorization decomposes the joint distribution of treatment assignment and potential outcomes into:
  • Identified components: These are based on the observed data, including the observed outcome distributions and the propensity score.
  • Unidentified components: These are the "selection functions" that describe how treatment assignment depends on unobserved potential outcomes. These are parameterized by sensitivity parameters.
  • Clean Separation: A key feature is that the observed data model (e.g., using Bayesian Additive Regression Trees or Dirichlet Process Mixtures) is fit independently of the sensitivity parameters. This means you can fit your primary model first and then perform sensitivity analysis post hoc.
  • Logistic Selection Specification: They propose a specific, interpretable parameterization for the unidentified selection functions using a logistic model. This makes the sensitivity parameters easier to understand and calibrate.
  • Calibration Heuristics: The paper provides methods for calibrating the magnitude of the sensitivity parameters by relating them to the variance explained by observed covariates, drawing an analogy to "implicit R-squared" measures.

Datasets

The paper uses two real-world datasets for demonstration:

  • NHANES data: Used to estimate the effect of taking two or more anti-hypertensives on average diastolic blood pressure. This example utilizes a nonparametric model for response surfaces with normally distributed residuals.
  • Job Training Data (JTPA): Used to analyze quantile treatment effects on zero-inflated income data. This example showcases the framework's ability to handle structured data and focuses on quantile treatment effects.

Key Insights and Ideas

  • Separation is Key: The most significant insight is the successful separation of model checking (how well a model fits the observed data) from sensitivity analysis (how robust results are to untestable assumptions). This is a major improvement over methods that conflate the two.
  • Interpretable Sensitivity: The proposed sensitivity parameters are designed to be interpretable, relating directly to how treatment assignment might depend on unobserved outcomes. This makes it easier for researchers to specify plausible ranges for these parameters.
  • Flexibility for Modern Models: The framework is designed to work seamlessly with modern, flexible statistical models (like BART and DPMs) that are increasingly used in causal inference. This is a crucial advantage over older methods that were tied to simpler parametric models.
  • "Causal Inference is Missing Data Twice": The authors highlight that causal inference with unmeasured confounding can be viewed as a double missing data problem, where both potential outcomes are missing for some units, and the selection mechanism into treatment is also unobserved.

Key Results

  • Demonstrated Flexibility: The framework was successfully applied to two distinct examples (NHANES and JTPA data) involving different outcome types (continuous, zero-inflated) and estimands (ATE, QTE).
  • Sensitivity of Results: The analyses showed that causal effect estimates (both ATE and QTE) can indeed be sensitive to the assumed unconfoundedness, with results changing meaningfully under plausible violations of the assumption.
  • Calibrated Sensitivity: The proposed calibration method, linking sensitivity parameters to the variance explained by observed covariates, provided a principled way to choose plausible values for the sensitivity parameters. For instance, in the NHANES example, they found that a sensitivity parameter magnitude comparable to the variance explained by BMI led to substantial shifts in the estimated ATE.

Limitations and Future Work

  • Outcome Overlap Condition: The validity of Tukey's factorization relies on an "outcome overlap condition," which assumes the support of missing potential outcomes is a subset of observed outcomes. While technically satisfied, the inferred missing data distribution can still be sensitive to the tails of the observed data density, especially when the distance between observed and missing data is large. This may become less plausible when covariates explain most of the outcome variance.
  • Covariate-Varying Sensitivity: The current framework assumes sensitivity parameters are independent of covariates. An extension to covariate-varying sensitivity parameters would increase the number of parameters and introduce challenges in calibration and reporting.
  • Multiple Treatments: Generalizing the approach to multiple or multi-level treatments is a promising avenue for future work.
  • Missing Data: Extending the framework to observational studies with missing data (in addition to unmeasured confounding) is another important direction.

Summary updated

Abstract

Abstract A fundamental challenge in observational causal inference is that assumptions about unconfoundedness are not testable from data. Assessing sensitivity to such assumptions is therefore important in practice. Unfortunately, some existing sensitivity analysis approaches inadvertently impose restrictions that are at odds with modern causal inference methods, which emphasize flexible models for observed data. To address this issue, we propose a framework that allows (1) flexible models for the observed data and (2) clean separation of the identified and unidentified parts of the sensitivity model. Our framework extends an approach from the missing data literature, known as Tukey’s factorization, to the causal inference setting. Under this factorization, we can represent the distributions of unobserved potential outcomes in terms of unidentified selection functions that posit a relationship between treatment assignment and unobserved potential outcomes. The sensitivity parameters in this framework are easily interpreted, and we provide heuristics for calibrating these parameters against observable quantities. We demonstrate the flexibility of this approach in two examples, where we estimate both average treatment effects and quantile treatment effects using Bayesian nonparametric models for the observed data.
BibTeX
@article{franks2019flexible,
  author = {Alexander Franks and Alexander D’Amour and Avi Feller},
  title = {{Flexible Sensitivity Analysis for Observational Studies Without Observable Implications}},
  journal = {Journal of the American Statistical Association},
  year = {2019},
  doi = {10.1080/01621459.2019.1604369},
}
Figure