Valid Causal Inference with (Some) Invalid Instruments

Jason S. Hartford, Victor Veitch, Dhanya Sridhar, Kevin Leyton-Brown

Abstract

Instrumental variable (IV) methods provide a powerful approach to estimating causal effects in the presence of unobserved confounding. But a key challenge when applying them is the reliance on untestable "exclusion" assumptions that rule out any relationship between the instrument variable and the response that is not mediated by the treatment. In this paper, we show how to perform consistent IV estimation despite violations of the exclusion assumption. In particular, we show that when one has multiple candidate instruments, only a majority of these candidates—or, more generally, the modal candidate-response relationship—needs to be valid to estimate the causal effect. Our approach uses an estimate of the modal prediction from an ensemble of instrumental variable estimators. The technique is simple to apply and is "black-box" in the sense that it may be used with any instrumental variable estimator as long as the treatment effect is identified for each valid instrument independently. As such, it is compatible with recent machine-learning based estimators that allow for the estimation of conditional average treatment effects (CATE) on complex, high dimensional data. Experimentally, we achieve accurate estimates of conditional average treatment effects using an ensemble of deep network-based estimators, including on a challenging simulated Mendelian Randomization problem.

Introduction

Causal inference is a cornerstone of data science, aiming to move beyond simple correlation to understand if a treatment $X$ truly causes an outcome $Y$. In the presence of unobserved confounders—hidden variables that influence both $X$ and $Y$—standard regression techniques fail. Researchers often turn to Instrumental Variables (IV) to resolve this. An ideal instrument is a variable that influences the treatment but has no direct effect on the outcome, a condition known as the "exclusion restriction."

However, in real-world domains like economics or genetics (e.g., Mendelian Randomization), the exclusion restriction is frequently violated and notoriously difficult to verify. Traditionally, if an instrument has a "side door" effect on the outcome, the resulting causal estimate is biased. This paper introduces ModeIV, a framework designed to produce consistent causal estimates even when a significant portion of candidate instruments are invalid.

The ModeIV Framework

ModeIV is a "black-box" meta-algorithm that can be wrapped around any existing IV estimator, such as DeepIV. The core intuition behind the method is Modal Validity. While invalid instruments may be biased in numerous, idiosyncratic ways, valid instruments should all converge toward the same true causal effect. As the authors suggest, "All happy instruments are alike; each unhappy instrument is unhappy in its own way."

Recovery of the target response curve using ModeIV.

The ModeIV process involves three primary steps:

  1. Ensemble Training: Train an ensemble of $k$ estimators, where each estimator utilizes a different candidate instrument.
  2. Modal Interval Selection: For a given test point, the algorithm analyzes predictions from all $k$ models to identify the "tightest cluster"—the smallest interval containing a pre-specified number ($V$) of estimates.
  3. Aggregation: The final estimate is produced by averaging the predictions within this modal cluster.

This approach allows ModeIV to move beyond "majority rule" requirements. While previous robust methods required over 50% of instruments to be valid, ModeIV only requires the valid instruments to be the plurality (the mode). If 30% are valid and the remaining 70% are biased in unique, non-overlapping ways, ModeIV can still recover the ground truth.

Experimental Evaluation

The authors evaluated ModeIV using two sophisticated simulation environments:

  • Biased Demand Simulation: A market simulation where price (treatment) and demand (outcome) are confounded. Bias parameters ($\gamma$) were introduced to create direct effects from instruments to outcomes, violating the exclusion restriction.
  • Mendelian Randomization (MR) Simulation: Modeled after genetic studies, using 100 candidate genetic markers (SNPs) as instruments. This task involved estimating Conditional Average Treatment Effects (CATE), where the causal effect varies based on individual characteristics, presenting a high-dimensional challenge for standard linear models.

Results and Discussion

ModeIV demonstrated significant robustness to bias. In the demand simulation, as the magnitude of exclusion violations increased, standard IV methods experienced skyrocketing error rates. In contrast, ModeIV maintained low error, closely tracking an "Oracle" model that possessed perfect knowledge of which instruments were valid.

Mean Squared Error (MSE) across varying levels of bias and invalid instrument counts.

In the Mendelian Randomization task, ModeIV successfully recovered both the sign and magnitude of treatment effects across sub-populations, even in low signal-to-noise scenarios where the treatment explained only 1-3% of outcome variance. The method also proved stable regarding the choice of $V$, provided the lower bound was not set excessively low.

Conclusion

ModeIV provides a flexible, high-performance solution for causal inference with potentially "polluted" data. By leveraging the power of clustering and ensembles, it enables the use of modern deep-learning estimators in settings where instrument validity cannot be guaranteed. Future work may focus on reducing the computational cost of training large ensembles and automatically identifying the set of valid instruments to further refine efficiency.

Summary updated

Abstract

Instrumental variable methods provide a powerful approach to estimating causal effects in the presence of unobserved confounding. But a key challenge when applying them is the reliance on untestable "exclusion" assumptions that rule out any relationship between the instrument variable and the response that is not mediated by the treatment. In this paper, we show how to perform consistent IV estimation despite violations of the exclusion assumption. In particular, we show that when one has multiple candidate instruments, only a majority of these candidates---or, more generally, the modal candidate-response relationship---needs to be valid to estimate the causal effect. Our approach uses an estimate of the modal prediction from an ensemble of instrumental variable estimators. The technique is simple to apply and is "black-box" in the sense that it may be used with any instrumental variable estimator as long as the treatment effect is identified for each valid instrument independently. As such, it is compatible with recent machine-learning based estimators that allow for the estimation of conditional average treatment effects (CATE) on complex, high dimensional data. Experimentally, we achieve accurate estimates of conditional average treatment effects using an ensemble of deep network-based estimators, including on a challenging simulated Mendelian Randomization problem.
BibTeX
@inproceedings{hartford2021valid,
  author = {Jason S. Hartford and Victor Veitch and Dhanya Sridhar and Kevin Leyton-Brown},
  title = {{Valid Causal Inference with (Some) Invalid Instruments}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=oCi0MZSitE},
}
Figure