Calibrating Sensitivity Parameters under Unobserved Confounding

38 concepts30 papers1 proposal

When evaluating policies or estimating causal effects from observational data, researchers often rely on the assumption of unconfoundedness—the idea that every variable influencing both actions and outcomes has been recorded. Because this assumption is frequently violated in real-world settings, sensitivity analysis has emerged as a vital tool. Instead of assuming zero hidden bias, researchers introduce a sensitivity parameter to bound the maximum possible strength of unobserved confounding.

This parameter defines what Greg Ver Steeg et al. term "causal ignorance intervals" in Ensembled Prediction Intervals for Causal Outcomes Under Hidden Confounding. The core intuition is straightforward: by varying this parameter, we can map out a range of worst-case and best-case scenarios.

However, this introduces a critical bottleneck: how do we select the right parameter value? As Nathan Kallus et al. demonstrate in Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning, if the chosen parameter is too low, the resulting intervals will fail to cover the true performance of the policy. Conversely, as noted by Mingzhang Yin et al. in Confounding-Robust Deferral Policy Learning, setting the parameter too high forces the model to be overly conservative, destroying its real-world utility.

How Sensitivity Parameters Constrain the Unobserved World

To understand why this framework works, we must look at how a mathematical parameter restricts an unobserved variable. In a typical sensitivity framework, the parameter acts as a strict "bias budget." For example, in Confounding-Robust Policy Improvement, Nathan Kallus et al. define uncertainty sets by bounding the maximum allowable deviation between observed treatment probabilities and true, unobserved probabilities.

Similarly, Angela Zhou et al. explain in Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders that these parameters constrain the influence of unobserved factors on action selection. Imagine an odds ratio bound of $\Gamma = 1.5$. This mathematically asserts that two individuals who look identical in the data cannot have their true odds of receiving a treatment differ by more than 50% due to unobserved factors. Once this budget is locked in, optimization algorithms can compute sharp, reliable bounds for conditional average treatment effects Interval Estimation of Individual-Level Causal Effects Under Unobserved Confounding. or worst-case policy performance Efficient and Sharp Off-Policy Learning under Unobserved Confounding.

The Problem of Unverifiability

The fundamental challenge of sensitivity analysis is that these parameters are entirely non-identifiable from the data. Because the confounders are unobserved, the dataset contains no direct evidence regarding their true magnitude. As David Bruns-Smith argues in Model-Free and Model-Based Policy Evaluation when Causality is Uncertain, there is no established mechanism to automatically derive or bound these parameters from historical datasets alone.

Consequently, selecting these bounds has traditionally been a subjective process. Researchers must rely heavily on domain expertise or arbitrary analyst heuristics, as seen in settings with outcomes truncated by death Identification and Estimation of Causal Effects with Outcomes Truncated by Death or variable importance estimation Doctor Rashomon and the UNIVERSE of Madness: Variable Importance with Unobserved Confounding and the Rashomon Effect. As Maresa Schröder et al. point out in Causal Fairness under Unobserved Confounding: A Neural Sensitivity Framework, while refutation strategies can test the robustness of guarantees, they cannot discover the true magnitude of hidden bias. If an expert misjudges this bound, it can completely undermine the reliability of statistical inferences in critical applications like digital clinical trials Electronic Medical Records Assisted Digital Clinical Trial Design.

Grounding Abstract Parameters in Observed Covariates

To resolve this subjectivity, a major direction in recent literature focuses on making sensitivity parameters interpretable. Abstract mathematical coefficients are notoriously difficult for human experts to evaluate. To ground these parameters, Victor Veitch et al. introduce a benchmarking approach in Sense and Sensitivity Analysis: Simple Post-Hoc Analysis of Bias Due to Unobserved Confounding. Instead of forcing experts to interpret abstract numbers, their framework allows analysts to compare the strength of a hypothetical hidden confounder directly against the strength of known, observed covariates. For instance, an expert can easily reason about a statement like: "Our conclusion holds unless there is an unobserved confounder twice as powerful as the patient's age."

Another powerful alternative is to map abstract bounds directly onto concrete, real-world phenomena. In Sensitivity analysis for causality in observational studies for regulatory science, Iván Díaz et al. demonstrate that translating sensitivity parameters into real-world metrics—such as spontaneous cure rates—allows domain experts to seamlessly integrate external clinical knowledge. Similarly, the use of specific, tangible metrics like odds ratios or mean ratios helps ground the analysis of latent groups in noncompliance settings Sensitivity analysis for principal ignorability violation in estimating complier and noncomplier average causal effects.

Towards Adaptive and Online Calibration

While grounding parameters in observed data makes offline evaluation more reliable, high-stakes autonomous systems require real-time adaptability. In control theory and reinforcement learning, fixing a static sensitivity parameter can be limiting. To address this, Myrl G Marmarelis et al. explore causal sensitivity analysis for model-predictive control in Off-policy Predictive Control with Causal Sensitivity Analysis. While their current framework focuses on offline calibration, the authors suggest that future systems could utilize online calibration techniques—such as multi-armed bandits—to dynamically adjust the sensitivity parameter during real-time operation. This transition from rigid, subjective guesses to interpretable benchmarks and adaptive tuning ensures that robustness guarantees remain both valid and practically useful.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sensitivity_parameter_selection_62d56a1a,
  title = {sensitivity parameter selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sensitivity-parameter-selection/rdr_62d56a1aed3f4f4f8721cf5c50a4a3c8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sensitivity_parameter_selection_62d56a1a_v1,
  title = {sensitivity parameter selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sensitivity-parameter-selection/rdr_62d56a1aed3f4f4f8721cf5c50a4a3c8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake