Many clinical questions about whether a treatment extends survival cannot be answered by randomized trials. Researchers fall back on instrumental variables: a factor that nudges patients toward a treatment but has no other route to the outcome. Physician preference, regional prescribing rates, random assignment with imperfect uptake, and genetic variants are common examples. Every such analysis rests on the "exclusion restriction," the assumption that the instrument affects survival only through the treatment. This assumption is normally checked once, for the whole cohort, if it is checked at all. Real cohorts are not uniform, though. A physician's preference for an aggressive regimen may be harmless for a robust patient. For a frail, high-risk patient the same preference may come bundled with other decisions, such as more intensive monitoring, earlier palliative referral, or different supportive care, and those decisions open a second path to the outcome. If that happens, the instrument is valid where patients are healthy and invalid where they are sick. The sick patients are exactly where real-world benefit diverges most from trial results: TrialTranslator found high-risk real-world phenotypes had median survival on treatment 62% lower than in the corresponding trials.
We propose to turn automated prognostic risk stratification into a tool for auditing instruments. A machine-learned survival risk score, fit on a held-out split using only baseline covariates, sorts patients into risk tiers. Within each tier we run falsification tests of the exclusion restriction adapted to right-censored survival outcomes. Right-censoring means some patients are still alive or lost to follow-up when the data end. The output is a validity map. It shows the tiers where the instrument survives falsification, the tiers where it fails, and the size of the implied direct effect where it fails. Causal estimates are reported only where the instrument holds, or alongside a sensitivity bound where it does not.
Core Claim
Instrument validity in clinical survival studies is plausibly heterogeneous along the axis of prognostic risk. A one-dimensional, honestly estimated risk score is the right lens for detecting this, for two reasons. First, it concentrates statistical power on the direction where clinical mechanisms predict violations. Second, it avoids the multiple-testing explosion of searching all covariate subgroups. The key identity is simple. Let $k$ index a risk tier and $\pi_k$ the share of patients in that tier whose treatment actually responds to the instrument (the compliance rate). Let $\tau_k$ be the tier's treatment effect on restricted mean survival time, which is the average survival time up to a fixed horizon. Let $\delta_k$ be any direct effect of the instrument on that same quantity. The instrument–outcome association in tier $k$ then decomposes as
$$\mathrm{RF}_k = \pi_k \tau_k + \delta_k ,$$
where $\mathrm{RF}_k$ is the "reduced-form" difference in restricted mean survival between instrument levels. Exclusion means $\delta_k = 0$. In patients whose predicted compliance is near zero, the first term vanishes, so any surviving instrument–outcome association is direct evidence of $\delta_k \neq 0$. When several candidate instruments are available, a second test applies: within a tier, valid instruments must agree on $\tau_k$, and disagreement signals invalidity. Neither test can be run tier-by-tier on censored data with current tools. Building both is the contribution.
Research Program
The work builds on four existing pieces and differs from each.
Data-driven exclusion criteria for instrumental variable studies supplies the method of learning a compliance score with sample splitting. It uses that score to drop low-compliance patients and gain power. We use the same score in the opposite way: predicted non-compliers become the falsification population. We also ask whether dropping them removes the high-risk tiers, a tension that paper never examines.
MISTR supplies the machinery for heavy censoring. It uses recursively imputed survival trees to fill in counterfactual event times, combined with Rubin's rules, which pool estimates across multiple imputations. Its instrumental-variable forest variant, MISTR-IV, assumes the instrument is valid. We reuse its imputation so that the falsification tests run on completed event times, and we put its validity assumption to the test.
The cross-instrument test (CAT) from Data-Driven Selection of Instrumental Variables for Additive Nonlinear, Constant Effects Models works as follows. It computes outcome residuals after removing the effect implied by one candidate instrument and checks, with distance correlation, whether those residuals are independent of the other candidates. CAT assumes a constant effect across the population, an assumption the research file flags as a key limitation. Applying it within risk tiers relaxes that assumption to "constant within tier." That is far more plausible clinically.
auton-survival supplies the prognostic models: Deep Cox Mixtures and Deep Survival Machines, which relax proportional hazards.
The concrete pipeline uses three data splits. On the first split, we fit a prognostic risk model on baseline covariates only, with treatment and instrument withheld so the score stays a pre-treatment quantity. We also fit a compliance model. On the second split, we impute censored event times using the MISTR approach. On the third split, which is the inference split, we form tiers from the frozen score. Within each tier we compute two things. The first is the zero-first-stage reduced-form test on restricted mean survival among patients predicted to be non-compliers. The second, when multiple instruments exist, is CAT on imputed survival times. Both tests are aggregated over imputations with Rubin's rules and corrected for the small, pre-specified number of tiers.
Where a tier fails, we report a bound on $\tau_k$ that uses the estimated $\delta_k$. The bound assumes the direct effect among compliers is no larger than some multiple of the direct effect measured in non-compliers, and we sweep that multiple as a sensitivity parameter. This is weaker than assuming exclusion. It is also more honest than discarding the tier, which is effectively what compliance-based exclusion does.
Causal inference and risk prediction of gestational diabetes mellitus based on case–control study and Mendel randomization is the closest prior bridge between these fields and motivates the project. It used genetic instruments only as an upfront filter to choose covariates for a risk score, without testing exclusion. We invert the relationship: the risk score audits the instrument.
Baselines
The naive reference point is a full-cohort instrumental-variable survival analysis that presumes the instrument is valid. We implement it with MISTR-IV on the same imputed data and the same forest hyperparameter budget. It answers the question "what does a practitioner report today?" No reported numerical result was found on Lacuna for MISTR-IV's estimation error. The paper reports only qualitative gains over Causal Survival Forests under 70–92% censoring, so we will regenerate its numbers on the shared semi-synthetic benchmark.
The strongest existing validity check is population-level CAT applied to imputed event times. We give it the same instruments, imputations, and distance-correlation test, but no tiering. It is the strongest competitor because it is the only data-driven, nonlinear instrument-selection test in the file with a necessary-and-sufficient guarantee. Beating it means higher power to detect a tier-localized violation at matched false-rejection rate.
The power-oriented competitor is the compliance-exclusion estimator of Liu et al.. On their Optum diabetes cohort it reduced standard error from 0.17 to 0.13 and raised power from 0.84 to 0.97. We run it with the identical compliance model and cutoff rule. Beating it means two things. First, retaining high-risk tier coverage that it discards. Second, smaller bias in the overall estimate whenever the injected violation sits in the tiers it keeps.
For weak-instrument heterogeneity we include the bias-correction estimator of Oprescu and Kallus, applied to restricted mean survival time computed from the imputed event times. They report a mean squared error of about 9.39 versus about 43.2 for a pure instrumental-variable estimator in a 50-dimensional simulation. It assumes exclusion everywhere, so it shows what happens when an untested violation is extrapolated across tiers.
Two ablations isolate the key ingredient. The first replaces prognostic tiers with tiers formed from the compliance score. The second uses random tiers of equal size. If prognostic tiering does not beat both on detection power when violations follow risk, the central claim fails. All methods share splits, imputations, and a fixed hyperparameter search budget.
Why not just…?
Why not just run a standard overidentification test or CAT on the whole cohort? A violation confined to the top risk tier is diluted by the valid majority. Pooling also forces a single effect size, so genuine effect heterogeneity can masquerade as invalidity or hide it. The semi-synthetic power curves in our experiments measure this directly. If population-level CAT matches tiered testing when violations are localized, the tiering adds nothing, and we will say so.
Why not just fit an instrumental-variable forest and inspect where estimates look strange? Under the decomposition above, a forest estimates $(\pi_k\tau_k + \delta_k)/\pi_k$. A large tier estimate could mean a large true effect or a direct effect divided by a small compliance rate. The forest cannot tell these apart. The zero-first-stage test can, because it isolates $\delta_k$ where $\pi_k \approx 0$. This alternative is a real threat when no tier contains near-zero compliers. Our experiments therefore vary the compliance distribution and report where the test loses power.
Why not just exclude low-compliance or high-risk patients, as in data-driven exclusion? This alternative buys precision by changing the target population. If non-compliance tracks frailty, the estimate then describes the healthy patients, for whom trial results already generalize. TrialTranslator found the survival-benefit shortfall versus trials was 5.7 months in high-risk patients and 2.3 months in low-risk patients. We will report the tier composition of the excluded set on every benchmark.
Why not skip instruments and use propensity adjustment? In auton-survival, propensity adjustment moved a regional hazard ratio from 0.728 to 0.890. That adjustment still cannot address unmeasured confounding. The gestational-diabetes study showed an observationally strong marker (HbA1c) losing its causal role under genetic instruments. Propensity adjustment is kept only as a descriptive reference.
Proposed Experiments
The primary testbed is semi-synthetic MIMIC-IV, built as in MISTR. It uses real ICU covariates, a simulated binary instrument, treatment, and survival times, with censoring swept from 30% to 92%. We inject a direct instrument-to-outcome effect $\delta$ of varying size into the top risk tier, into a covariate-defined subgroup unrelated to risk (a negative control for the "risk is the right axis" claim), or nowhere. We report the following on every condition: false rejection rate when the instrument is valid, detection power, validity-map accuracy, and bias and coverage of tier effects.
The second testbed is ACTG 175, an HIV trial in which randomization is a valid instrument by design. We inject noncompliance and, separately, a risk-dependent direct effect. The decisive check here is that the procedure does not reject when no violation is injected under heavy artificial censoring.
The Illinois unemployment-insurance experiment provides a real randomized instrument for time-to-employment. Here the procedure should certify validity across all tiers. Rejection would indicate the tests are miscalibrated.
For an observational application, we use SEER breast cancer. Candidate instruments are area-level treatment rates and registry region, building on the California–Louisiana contrast analyzed in auton-survival. We use CAT across these multiple candidates within tiers and report the validity map. We also report how much of the high-risk tier compliance exclusion would discard.
The project's central claim is confirmed if three things hold. Tiered testing must beat population-level CAT and random-tier ablations in power for risk-localized violations at nominal false-rejection rates. It must not beat them for violations unrelated to risk. Bounded tier estimates must cover the true effect where presumed-valid MISTR-IV and the Oprescu–Kallus estimator do not. If prognostic tiers are no better than compliance tiers or random tiers, the conclusion is that within-subgroup instrument auditing is useful but risk stratification is not its natural axis. That result would also be worth reporting.
References
- Evaluating generalizability of oncology trial results to real-world patients using machine learning-based trial emulations
- Data-driven exclusion criteria for instrumental variable studies
- Heterogeneous Treatment Effect in Time-to-Event Outcomes: Harnessing Censored Data with Recursively Imputed Trees
- Data-Driven Selection of Instrumental Variables for Additive Nonlinear, Constant Effects Models
- auton-survival: an Open-Source Package for Regression, Counterfactual Estimation, Evaluation and Phenotyping with Censored Time-to-Event Data
- Causal inference and risk prediction of gestational diabetes mellitus based on case–control study and Mendel randomization
- Estimating Heterogeneous Treatment Effects by Combining Weak Instruments and Observational Data