A retrospective cohort study estimates a treatment effect from records that were collected during routine care. Epidemiologists do this, for example, when they ask whether intubating COVID-19 patients later raised their risk of death. Before any estimate exists, someone has to make dozens of small choices. How many hours of data make up the observation window? At what percentile are outlier lab values clipped? How are missing vitals filled in? When exactly does "treated" begin? And the machine-learning models that adjust for confounding each need their own settings: tree depth, regularization strength, matching caliper. Each of these choices is a hyperparameter, meaning a setting the analyst picks rather than one the data determine. The confidence interval a paper reports counts only sampling noise. It says nothing about the spread that would appear if a different, equally defensible analyst had made different choices.
This note proposes to measure that hidden spread on real multi-hospital intensive care unit (ICU) records, and to test which rule for choosing among defensible settings actually recovers a known effect. The approach plants treatment effects of known size into real ICU records at the raw, pre-cleaning stage. That way the full cleaning pipeline, with all its messiness, sits between the data and the estimate, and there is still a ground truth to score against. It also uses negative-control outcomes, which are outcomes the treatment cannot plausibly affect, as a filter for bad configurations that needs no ground truth.
Core Claim
We claim that in ICU retrospective cohorts, the variation in effect estimates across defensible pipeline configurations is comparable to or larger than the reported sampling standard error. We also claim that most of this variation comes from preprocessing choices, not from the tuning of the adjustment models. If both hold, the standard remedy of tuning the adjustment models well is necessary but addresses the smaller part of the problem.
The proposed remedy is simple. First, run a grid of defensible configurations. Discard those whose estimates on negative-control outcomes sit far from zero. Among the survivors, rank by a doubly robust surrogate score. Finally, report an interval widened by the variation that remains among the retained configurations. The claim is falsifiable: on planted-effect data, this rule must beat three alternatives on bias and on coverage, meaning how often the interval contains the true effect. The alternatives are defaults, selection by prediction loss, and a uniform average over all configurations. If it does not beat them, the simpler rule is what we recommend.
What Prior Work Establishes and Leaves Open
Two findings motivate the gap, and they have never been put together. Yet Another ICU Benchmark (YAIB) harmonized five ICU databases: MIMIC-III, MIMIC-IV, eICU, HiRID and AUMCdb. It showed that preprocessing decisions, such as exclusion filters, observation windows, clipping thresholds, imputation, and label definitions, move predictive AUROC more than switching from logistic regression to Transformers. It measured only prediction quality, never a treatment effect. Separately, Hyperparameter Tuning for Causal Inference with Double Machine Learning studied double machine learning, which estimates an effect by fitting separate models for the outcome and for treatment assignment and combining them. It showed that default settings for these models produce severe bias. It found that tuning on the full sample or within the estimation folds works well, while splitting off a separate tuning sample wastes data. It also found that outcome prediction loss picks the better causal model form in over 75% of heterogeneous-effect settings. Its data were synthetic or the semi-synthetic IHDP benchmark, and the preprocessing was held fixed. We ask what YAIB's preprocessing sensitivity does to the kind of causal estimate Bach et al. studied.
Two further papers supply methods we reuse and test. Empirical Analysis of Model Selection for Heterogenous Causal Effect Estimation benchmarked 34 surrogate metrics for choosing causal models without counterfactual ground truth. It found that doubly robust and TMLE-based T-scores work best, and that AutoML tuning of the adjustment models helps even simple metrics. All of this was on ACIC 2016 and on flow-generated RealCause data. We borrow the doubly robust surrogate and ask whether it still works when the variation comes from upstream data cleaning rather than from the choice of estimator. Tuning-parameter-free optimal propensity score matching showed a concrete case of the problem. On LaLonde, the estimated effect from ordinary propensity score matching moves from 514.42 to nearly 900 as the number of matches $M$ goes from 3 to 15. Their monotone estimator, fitted with the pool-adjacent-violators algorithm (PAVA), stays near 915. Their fix removes one matching setting. It assumes monotone single-index structure and leaves every preprocessing choice untouched, so it serves as a baseline rather than a competitor for the full problem.
Two papers constrain the design. Looking for Out-of-Distribution Environments in Critical Care found that apparent hospital-level shift in eICU is often a small-sample effect. Our cross-hospital analysis therefore groups hospitals so that each unit is large enough for its variability to reflect real differences rather than noise. The same paper found that smaller models transfer better, so the model grid includes low-capacity options. DeepComBat is precedent for engineering robustness to settings, rather than tuning them away, in multi-site retrospective imaging cohorts. It works on imaging batch effects, not on treatment confounding.
Research Program
The clinical question is the one Timing to intubation COVID-19 patients studied in a single São Paulo center, where it reported an adjusted hazard ratio of 1.018 per day of delay. We transpose it to early versus delayed invasive ventilation in general ICU patients with respiratory failure, in MIMIC-IV and eICU. The question fits well because the treatment definition itself is a preprocessing setting: "early" can be measured from admission, from first low oxygen saturation, or from first high oxygen requirement, with varying cutoffs.
We will build a configuration space along two layers. The preprocessing layer covers the observation window, aggregation bin width, outlier clipping percentile, imputation method (carry-forward, median, or model-based), the treatment-onset anchor and cutoff, and the cohort exclusion rules. The estimation layer covers the estimator family (propensity matching with varying $M$ and caliper, inverse weighting, double machine learning in its partially linear and interactive forms), the adjustment-model family (logistic or lasso, gradient boosting, a small recurrent network), and the tuning protocol (defaults, full-sample, on-folds). Sampling a few hundred configurations from this space, restricted to ones a reviewer at a clinical journal would accept, gives the "defensible" set.
Planted ground truth works as follows. We keep the real raw hourly vitals, labs, and real treatment times, and we simulate 28-day mortality from the raw streams with a fixed function. That function is fitted to real outcomes so that it looks realistic. It includes a known treatment effect, effect heterogeneity, and a confounder drawn from a raw signal that some preprocessing choices smooth away or clip. The difference from RealCause-style benchmarks, including those used by Mahajan et al., is that the simulated outcome sits upstream of cleaning. Preprocessing choices can therefore help or hurt recovery of the truth, which is exactly the mechanism under study.
The negative-control filter needs outcomes the treatment cannot cause but which share its confounding. Examples are pre-admission chronic diagnoses recorded later in the stay, and admission-day lab abnormalities measured before the treatment decision. A configuration whose estimated "effect" on these outcomes is far from zero is adjusting badly. This filter runs on real outcomes with no simulation, so it is usable in practice.
The main measured quantity is the choice-to-sampling ratio, $\rho = \sigma_{\text{between}} / \bar{s}_{\text{within}}$. Here $\sigma_{\text{between}}$ is the standard deviation of point estimates across defensible configurations, and $\bar{s}_{\text{within}}$ is their average reported standard error. A value of $\rho \geq 1$ means the published interval understates uncertainty by at least a factor of $\sqrt{2}$ once the two components are added in quadrature. We will also decompose the variance with a two-way random-effects model with preprocessing and estimation as the factors. That tells us which layer dominates.
Baselines
The naive reference point is what most retrospective studies do. It uses the YAIB default extraction, logistic-regression propensity scores, 1:1 nearest-neighbor matching, and library-default settings, which is the style of analysis in Long-Term Survival in Men With High Grade Prostate Cancer. It runs on the same cohorts. Beating it on planted-effect bias is expected and would carry little weight on its own. Its role is to anchor $\rho$ and to show how far a typical published estimate falls from the truth.
The strongest existing method is double machine learning with AutoML-tuned adjustment models (FLAML), tuned on the full sample, following Bach et al.. Preprocessing is chosen by best cross-validated outcome prediction loss, the natural extension of their finding that prediction loss selects the right causal form. For this pipeline on IHDP the file reports only the >75% correct model-form selection rate. There is no reported result found on Lacuna for its bias or coverage on MIMIC-IV or eICU, and producing that number is part of the contribution. For fairness, it gets the same configuration pool, the same total compute measured in adjustment-model fits, and the same FLAML time budget per fit as our rule. Beating it means lower absolute bias and coverage closer to 95% on planted effects, across at least three planted effect sizes, including zero. A tie is also possible, and it would be a useful finding.
The doubly robust T-score selection of Mahajan et al., who report that two-level selection improves top-model identification in 28.7% of benchmarks, is run over the same pool without the negative-control filter. The PAVA matching estimator of Liu and Qin is run under each preprocessing configuration. That shows whether removing the matching settings removes much of $\rho$. On LaLonde it stayed near 915 while fixed-$M$ matching ranged from 514.42 to nearly 900.
Two ablations isolate the key ingredients. The first uses the negative-control filter alone, with no surrogate ranking. The second uses uniform averaging over all defensible configurations with a widened interval but no filtering. The second ablation is effectively a multiverse analysis. If it matches our full rule, the filtering step adds nothing.
Why not just…?
Why not just use sensible defaults and a standard extraction? Bach et al. already show that defaults bias double-machine-learning estimates on data where the answer is known. Our planted-effect runs will quantify the same failure on real ICU covariates. The naive baseline gives the number directly.
Why not just tune everything on predictive loss with AutoML, as YAIB and Bach et al. do? This is the most serious threat, because Bach et al. find that prediction loss picks the right model form most of the time. The concern is specific to preprocessing. A clipping or smoothing choice that removes a confounding signal can improve outcome prediction while worsening confounding adjustment, and prediction loss cannot see that. The planted confounder is built to test exactly this. If prediction-loss selection matches our rule on bias and coverage, we will report that it suffices and drop the more elaborate rule.
Why not just run a multiverse (specification-curve) analysis and report the spread? A multiverse average treats every defensible configuration as equally credible. It reports spread but cannot say which configurations are wrong. The uniform-averaging ablation is precisely this alternative. It wins if negative-control filtering fails to move estimates toward the planted truth. Filtering is only worth its cost if it shrinks bias while keeping coverage.
Why not just use a tuning-free estimator like PAVA matching? It removes the choice of $M$ and the number of subclasses. It does nothing about windows, imputation, or treatment definitions, which YAIB suggests matter most. The variance decomposition with PAVA held fixed measures how much of $\rho$ survives.
Proposed Experiments
The first decisive check is whether $\rho$ exceeds 1 on real MIMIC-IV and eICU outcomes for the intubation-timing question. We will also check whether the preprocessing factor explains more variance than the estimation factor. If $\rho$ is well below 1, the problem is smaller than claimed, and the note's main recommendation reduces to "tune the adjustment models."
The second check uses planted effects on MIMIC-IV with sizes of zero, small, and moderate. It compares bias, root mean squared error, and 95% interval coverage across the naive rule, prediction-loss selection, DR T-score selection, PAVA, the uniform multiverse, and our filtered-and-widened rule, all at matched compute. We also record total fits, because sweep cost is part of the true price of the analysis.
The third check tests transport. Selection rules are fitted on MIMIC-IV, a single center, and applied to eICU hospitals grouped by region, with each group large enough to avoid the small-sample artifacts Spathis and Hyland describe. We then measure whether each rule's chosen configuration gives estimates that agree across regions on real outcomes, and whether it keeps negative-control estimates near zero.
The fourth check guards against circularity. We rerun the planted-effect study with a confounder that the negative controls do not share. This shows how much of the filter's advantage depends on the negative controls being well chosen, a condition an analyst cannot verify in practice.
References
- Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML
- Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study
- Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation
- Tuning-parameter-free optimal propensity score matching approach for causal inference
- Looking for Out-of-Distribution Environments in Critical Care: A case study with the eICU Database
- DeepComBat: A Statistically Motivated, Hyperparameter-Robust, Deep Learning Approach to Harmonization of Neuroimaging Data
- Timing to intubation COVID-19 patients: can we put it off until tomorrow?
- Long-Term Survival in Men With High Grade Prostate Cancer: A Comparison Between Conservative Treatment, Radiation Therapy and Radical Prostatectomy—A Propensity Scoring Approach