Sensitivity Analysis for Causal ML: A Use Case at Booking.com

Philipp Bach, Victor Chernozhukov, Carlos Cinelli, Lin Jia, Sven Klaaßen, Nils Skotara, Martin SpindlerarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces sensitivity analysis for Causal Machine Learning (CML) to assess the robustness of causal effect estimates derived from observational data. It demonstrates a practical application of a new non-parametric method for bounding omitted variable bias using a simulated use case from Booking.com, highlighting its importance for real-world decision-making.

What Problem Does This Paper Address?

Estimating causal effects from observational data is crucial when randomized controlled trials (RCTs) are infeasible. However, observational studies rely on untestable assumptions, such as the absence of unobserved confounders (variables that influence both treatment and outcome). If these assumptions are violated, causal estimates can be biased, leading to incorrect conclusions. This paper addresses the critical need to quantify how much these estimates might be affected by potential unobserved confounders, a problem often overlooked in practice.

Methods

The paper leverages the framework of Causal Machine Learning (CML), specifically employing Double/Debiased Machine Learning (DML) for estimation. DML is a robust approach that combines high-quality machine learning estimators with specific techniques (Neyman orthogonal scores and sample splitting) to provide reliable causal effect estimates and valid statistical inference, even with complex data.

The core contribution is the application of a sensitivity analysis method proposed by Chernozhukov et al. (2023). This method provides general, non-parametric bounds on the bias introduced by omitted variables. It quantifies the potential impact of unobserved confounders using sensitivity parameters that measure their correlation with the outcome and treatment assignment. The paper focuses on the Average Treatment Effect on the Treated (ATT), a common parameter of interest in industry.

Datasets

The paper uses a simulated dataset inspired by a real use case at Booking.com. This simulated data is designed to mimic a scenario where a customer's decision to book an ancillary product (treatment) might influence their subsequent accommodation bookings (outcome), with potential unobserved confounders like customer loyalty. The authors emphasize that these are not actual Booking.com results but a reproducible example.

Key Insights and Ideas

  • The "Gold Standard" isn't always feasible: While RCTs are ideal, real-world constraints often force reliance on observational data, making sensitivity analysis indispensable.
  • Unobserved confounders are a persistent threat: Even with sophisticated CML techniques, the possibility of unmeasured variables influencing results means we can't be certain about causal claims without sensitivity analysis.
  • Quantifying uncertainty is key: Sensitivity analysis moves beyond simply stating "unobserved confounders might exist" to providing concrete bounds on how much bias they could introduce.
  • CML and Sensitivity Analysis are natural partners: The paper demonstrates how modern CML tools (like DML) can be seamlessly integrated with advanced sensitivity analysis methods.
  • Benchmarking provides context: Relating potential unobserved confounders to the strength of observed confounders (like membership variables) offers a practical way to assess plausibility and interpret sensitivity results.

Key Results

The paper demonstrates the application of the sensitivity analysis framework on the simulated Booking.com use case. While specific numerical results are based on simulated data, the key takeaway is that the estimated ATT (positive and significant) remains robust to plausible levels of confounding, as determined by benchmarking against observed variables. The analysis provides quantitative bounds on the ATT under various confounding scenarios, showing how these bounds change and illustrating the importance of considering these scenarios for drawing reliable conclusions.

Limitations and Future Work

The authors acknowledge that the use case is presented in a stylized way and that the empirical analysis is based on simulated data due to confidentiality concerns. They also note that while the framework is general, the interpretation of sensitivity parameters can be refined with additional parametric assumptions. Future work could involve exploring other sensitivity approaches and further integrating these methods into practical workflows. The paper implicitly suggests that broader adoption of sensitivity analysis in applied research is a key future direction.

Summary updated

Abstract

Causal Machine Learning has emerged as a powerful tool for flexibly estimating causal effects from observational data in both industry and academia. However, causal inference from observational data relies on untestable assumptions about the data-generating process, such as the absence of unobserved confounders. When these assumptions are violated, causal effect estimates may become biased, undermining the validity of research findings. In these contexts, sensitivity analysis plays a crucial role, by enabling data scientists to assess the robustness of their findings to plausible violations of unconfoundedness. This paper introduces sensitivity analysis and demonstrates its practical relevance through a (simulated) data example based on a use case at Booking.com. We focus our presentation on a recently proposed method by Chernozhukov et al. (2023), which derives general non-parametric bounds on biases due to omitted variables, and is fully compatible with (though not limited to) modern inferential tools of Causal Machine Learning. By presenting this use case, we aim to raise awareness of sensitivity analysis and highlight its importance in real-world scenarios.
BibTeX
@misc{bach2025sensitivity,
  author = {Philipp Bach and Victor Chernozhukov and Carlos Cinelli and Lin Jia and Sven Klaaßen and Nils Skotara and Martin Spindler},
  title = {{Sensitivity Analysis for Causal ML: A Use Case at Booking.com}},
  year = {2025},
  eprint = {2510.09109},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.09109},
  howpublished = {arXiv},
}
Figure