Flexible sensitivity analysis for observational studies without\n observable implications

Alexander Franks, Alexander D’Amour, Avi Feller

Abstract

A fundamental challenge in observational causal inference is that assumptions about unconfoundedness are not testable from data. Assessing sensitivity to such assumptions is therefore important in practice. Unfortunately, some existing sensitivity analysis approaches inadvertently impose restrictions that are at odds with modern causal inference methods, which emphasize flexible models for observed data. To address this issue, we propose a framework that allows (1) flexible models for the observed data and (2) clean separation of the identified and unidentified parts of the sensitivity model. Our framework extends an approach from the missing data literature, known as Tukey's factorization, to the causal inference setting. Under this factorization, we can represent the distributions of unobserved potential outcomes in terms of unidentified selection functions that posit an unidentified relationship between the treatment assignment indicator and the observed potential outcomes. The sensitivity parameters in this framework are easily interpreted, and we provide heuristics for calibrating these parameters against observable quantities. We demonstrate the flexibility of this approach in two examples, where we estimate both average treatment effects and quantile treatment effects using Bayesian nonparametric models for the observed data.

Introduction

In observational studies, researchers often seek to estimate the causal effect of a treatment $T$ on an outcome $Y$. The standard approach relies on the assumption of "unconfoundedness"—the belief that all variables influencing both treatment and outcome have been measured. However, this assumption is inherently untestable; one can never definitively prove the absence of a hidden confounder.

Sensitivity analysis serves as a critical tool to address this uncertainty by asking: "How strong would a hidden bias have to be to change my qualitative conclusions?" Despite its importance, many traditional sensitivity methods are "brittle," requiring simple parametric models that may not fit the data well. When researchers attempt to use flexible, modern machine learning models, these traditional tools often impose hidden restrictions or require computationally prohibitive re-fitting for every level of hypothesized bias.

Methodology: Tukey’s Factorization and Selection Functions

The authors propose a framework that achieves a "clean separation" between what the data reveals (the identified part) and what must be assumed (the unidentified part). This is achieved by adapting Tukey’s factorization from the missing data literature to the causal setting. Causal inference is framed as a "missing data twice" problem: we never observe the control outcome for the treated group, nor the treated outcome for the control group.

By parameterizing bias through a "selection function," the researchers can describe the relationship between treatment assignment and unobserved potential outcomes. Specifically, they focus on logistic-mEF models (logistic selection with mixtures of exponential families), which are compatible with powerful Bayesian nonparametric tools like Bayesian Additive Regression Trees (BART) and Dirichlet Process Mixtures (DPM).

Sensitivity of potential outcome distributions to selection parameters $\Delta \Psi_T$ and $\Delta \Psi_Y$.

A key advantage of this factorization is computational efficiency. Because the identified and unidentified components are decoupled, researchers only need to fit their flexible model to the observed data once. Sensitivity analysis is then performed post hoc by "tilting" the results using sensitivity parameters. As shown in the figure above, varying these parameters allows researchers to visualize how the estimated distributions of potential outcomes shift under different assumptions of hidden confounding.

Empirical Applications

The framework's flexibility is demonstrated through two distinct real-world examples:

  1. NHANES III (National Health and Nutrition Examination Survey): Estimating the effect of diuretic medication on blood pressure. While the initial analysis suggested a non-significant effect, the sensitivity analysis revealed that a hidden confounder with an effect size comparable to BMI could significantly alter the results.
  2. JTPA (Job Training Partnership Act): Analyzing the impact of job training on earnings. This dataset is "zero-inflated" (many participants have zero earnings), a challenge that standard causal models struggle to handle. The proposed method successfully modeled this complexity and showed that results were highly sensitive to assumptions about participant selection.

To make these results more interpretable, the authors provide a "calibration by analogy" heuristic. This allows researchers to map abstract sensitivity parameters to an "implicit $R^2$," enabling statements such as: "If the unobserved confounder explains as much variance as 'Age,' my treatment effect would vanish."

Key Insights and Conclusion

The primary contribution of this work is the decoupling of flexible modeling from rigorous sensitivity analysis. This approach prevents "data snooping," where sensitivity assumptions might inadvertently influence the initial model fit. Furthermore, the authors demonstrate that for common targets like the Average Treatment Effect (ATE) or Quantile Treatment Effect (QTE), modeling the correlation between potential outcomes (the "copula") is unnecessary, greatly simplifying the analysis.

While the method currently assumes sensitivity parameters are constant across covariates and focuses on binary treatments, it provides a robust foundation for modern, flexible causal inference that does not sacrifice the rigor of sensitivity testing.

Summary updated

Abstract

A fundamental challenge in observational causal inference is that assumptions\nabout unconfoundedness are not testable from data. Assessing sensitivity to\nsuch assumptions is therefore important in practice. Unfortunately, some\nexisting sensitivity analysis approaches inadvertently impose restrictions that\nare at odds with modern causal inference methods, which emphasize flexible\nmodels for observed data. To address this issue, we propose a framework that\nallows (1) flexible models for the observed data and (2) clean separation of\nthe identified and unidentified parts of the sensitivity model. Our framework\nextends an approach from the missing data literature, known as Tukey's\nfactorization, to the causal inference setting. Under this factorization, we\ncan represent the distributions of unobserved potential outcomes in terms of\nunidentified selection functions that posit an unidentified relationship\nbetween the treatment assignment indicator and the observed potential outcomes.\nThe sensitivity parameters in this framework are easily interpreted, and we\nprovide heuristics for calibrating these parameters against observable\nquantities. We demonstrate the flexibility of this approach in two examples,\nwhere we estimate both average treatment effects and quantile treatment effects\nusing Bayesian nonparametric models for the observed data.\n
BibTeX
@article{franks2019flexible,
  author = {Alexander Franks and Alexander D’Amour and Avi Feller},
  title = {{Flexible sensitivity analysis for observational studies without\textbackslash\{\}n observable implications}},
  journal = {Journal of the American Statistical Association},
  year = {2019},
  doi = {10.1080/01621459.2019.1604369},
}
Figure