TL;DR
This paper introduces sensitivity analysis for Causal Machine Learning (CML) to assess the robustness of causal effect estimates derived from observational data. It demonstrates a practical application of a new non-parametric method for bounding omitted variable bias using a simulated use case from Booking.com, highlighting its importance for real-world decision-making.
What Problem Does This Paper Address?
Estimating causal effects from observational data is crucial when randomized controlled trials (RCTs) are infeasible. However, observational studies rely on untestable assumptions, such as the absence of unobserved confounders (variables that influence both treatment and outcome). If these assumptions are violated, causal estimates can be biased, leading to incorrect conclusions. This paper addresses the critical need to quantify how much these estimates might be affected by potential unobserved confounders, a problem often overlooked in practice.
Methods
The paper leverages the framework of Causal Machine Learning (CML), specifically employing Double/Debiased Machine Learning (DML) for estimation. DML is a robust approach that combines high-quality machine learning estimators with specific techniques (Neyman orthogonal scores and sample splitting) to provide reliable causal effect estimates and valid statistical inference, even with complex data.
The core contribution is the application of a sensitivity analysis method proposed by Chernozhukov et al. (2023). This method provides general, non-parametric bounds on the bias introduced by omitted variables. It quantifies the potential impact of unobserved confounders using sensitivity parameters that measure their correlation with the outcome and treatment assignment. The paper focuses on the Average Treatment Effect on the Treated (ATT), a common parameter of interest in industry.
Datasets
The paper uses a simulated dataset inspired by a real use case at Booking.com. This simulated data is designed to mimic a scenario where a customer's decision to book an ancillary product (treatment) might influence their subsequent accommodation bookings (outcome), with potential unobserved confounders like customer loyalty. The authors emphasize that these are not actual Booking.com results but a reproducible example.
Key Insights and Ideas
- The "Gold Standard" isn't always feasible: While RCTs are ideal, real-world constraints often force reliance on observational data, making sensitivity analysis indispensable.
- Unobserved confounders are a persistent threat: Even with sophisticated CML techniques, the possibility of unmeasured variables influencing results means we can't be certain about causal claims without sensitivity analysis.
- Quantifying uncertainty is key: Sensitivity analysis moves beyond simply stating "unobserved confounders might exist" to providing concrete bounds on how much bias they could introduce.
- CML and Sensitivity Analysis are natural partners: The paper demonstrates how modern CML tools (like DML) can be seamlessly integrated with advanced sensitivity analysis methods.
- Benchmarking provides context: Relating potential unobserved confounders to the strength of observed confounders (like membership variables) offers a practical way to assess plausibility and interpret sensitivity results.
Key Results
The paper demonstrates the application of the sensitivity analysis framework on the simulated Booking.com use case. While specific numerical results are based on simulated data, the key takeaway is that the estimated ATT (positive and significant) remains robust to plausible levels of confounding, as determined by benchmarking against observed variables. The analysis provides quantitative bounds on the ATT under various confounding scenarios, showing how these bounds change and illustrating the importance of considering these scenarios for drawing reliable conclusions.
Limitations and Future Work
The authors acknowledge that the use case is presented in a stylized way and that the empirical analysis is based on simulated data due to confidentiality concerns. They also note that while the framework is general, the interpretation of sensitivity parameters can be refined with additional parametric assumptions. Future work could involve exploring other sensitivity approaches and further integrating these methods into practical workflows. The paper implicitly suggests that broader adoption of sensitivity analysis in applied research is a key future direction.
Summary updated