Choosing Treatment-Effect Models for Radiotherapy-Plus-Immunotherapy Cohorts When Censoring Breaks the Usual Checks

Adding radiation to immune checkpoint drugs helps some cancer patients a great deal and others not at all. The proposed explanation is that radiation kills tumor cells in a way that alerts the immune system, which turns an immunologically "cold" tumor into one the drugs can act on. To decide who should get the extra radiation, clinicians need per-patient estimates of benefit. Machine learning now offers dozens of estimators for this from survival data, and they disagree. Because no patient is observed both with and without radiation, the usual practice of picking the model with the best held-out accuracy does not work. The checks that partly fix this have a known weakness. Doubly robust "pseudo-outcome" criteria build a noisy stand-in for each patient's true benefit, and they are expected to become very noisy when most follow-up is cut short. Heavy censoring of this kind is typical of retrospective oncology cohorts.

This note proposes that the choice of validation criterion, and not only the choice of estimator, should depend on measurable properties of the cohort: how much follow-up is censored, how well treated and untreated patients overlap, and how much the candidate estimators disagree. We will build a censoring-aware doubly robust criterion for restricted mean survival time. Restricted mean survival time is the average event-free time up to a fixed horizon. We will then train a small selector that reads these cohort diagnostics and decides which criterion to trust. Both will be tested on simulated survival data whose treatment effect is shaped like immune priming: a threshold, multiplicative benefit concentrated in low-infiltration tumors, built on real clinical covariates. The claim is falsifiable and simple. A single fixed criterion should be near-optimal at low censoring and should fail above roughly 70% censoring. A diagnostic-conditioned choice should close most of that gap.

Why this gap is real

Three results set up the problem. In Search of Insights, Not Magic Bullets compares validation criteria for conditional average treatment effect (CATE) models, meaning per-patient treatment benefit as a function of covariates. It finds three things: choosing by factual prediction error is a trap; pseudo-outcome criteria built on doubly robust or R-learner losses are the least bad option; and "plug-in" criteria favor candidates that share their own inductive bias, which the authors call congeniality bias. That study covers only uncensored outcomes. SurvHTE-Bench runs 53 survival treatment-effect estimators across 52 datasets. It reports no universal winner. Simple imputation-based learners do well at low censoring, while survival-aware meta-learners and direct survival models take over above about 70% censoring and are more stable there. It also evaluates estimators against simulated ground truth, which a practitioner never has. Reliable Selection of Heterogeneous Treatment Effect Estimators treats selection as multiple testing with familywise error control. It is limited to uncensored benchmarks such as ACIC 2016, IHDP and Twins. Taken together, these papers motivate the proposal. Estimator rankings shift with censoring, the best known criteria were never tested under censoring, and the one error-controlled selector ignores censoring.

The clinical side explains why this matters for radiotherapy priming specifically. In esophageal cancer, Zhang et al. found no overall survival benefit from postoperative radiotherapy in unselected patients (median 38 and 39 months in the two cohorts). A benefit appeared only when radiotherapy was combined with immunotherapy in the squamous subgroup (p = 0.044), and the analysis used propensity matching and Cox models, which cannot resolve individual heterogeneity. In lung cancer, the meta-analysis by Liu, Xu et al. reports pooled benefits (overall survival HR 0.72) alongside a 76% higher pneumonia risk, together with unpredictable responses at distant, non-irradiated lesions (the abscopal effect). The mechanism papers suggest the effect has a particular form. Corn et al. show radiation reshaping macrophage-rich, T-cell-excluded regions when activated CD8+ cells start near 0–5%. Xu et al. cure distant tumors in half of treated mice with 8 Gy × 3 plus anti-PD-1. These findings imply the benefit is thresholded and multiplicative in baseline immune infiltration, not additive. That matters for model selection. On Inductive Biases for Heterogeneous Treatment Effect Estimation shows that hard-coding additive structure helps when effects are simple and hurts when they are complex, so the criteria may be most biased exactly where priming biology lives.

Core Claim

The reliability of survival treatment-effect validation criteria depends on the cohort's regime, and that regime can be read from observable diagnostics. Specifically, an inverse-probability-of-censoring-weighted doubly robust criterion will rank estimators well when censoring is moderate. Above roughly 70% censoring its variance will swamp its bias advantage, and a plug-in criterion built on a survival-aware model will rank estimators better despite congeniality bias. A selector conditioned on censoring fraction, propensity overlap, Kaplan–Meier tail mass at the horizon, and the spread of candidate estimates should match the better fixed criterion in every regime. No fixed criterion should do that. This differs from SurvHTE-Bench, which selects with oracle ground truth. It also differs from Curth et al. and Guo & Gao, whose criteria assume no censoring and are applied in the same way regardless of regime.

Research Program

The first component is the criterion. For a horizon $L$, each candidate estimator $k$ outputs $\hat\tau_k(x)$, its estimated difference in restricted mean survival time between radiotherapy-plus-immunotherapy and immunotherapy alone. We will build a doubly robust pseudo-outcome for that difference in the standard way. It combines an outcome model's predicted survival time, a propensity correction, and an inverse weight for the probability of remaining uncensored until $\min(T, L)$. Here $T$ is the observed event time and the censoring model is fit on held-out folds. Fitting nuisance models on held-out folds follows the two-layer cross-fitting of Guo & Gao, which keeps the nuisance estimators separate from the selection step. We will also test truncating the censoring weights. This is the obvious variance fix, and it trades bias for variance, so it needs its own evaluation.

The second component is the testbed. We will reuse SurvHTE-Bench's semi-synthetic covariates (ACTG 175 and MIMIC-IV) and its Twins setup. On top of these we will add a family of priming-shaped outcome models. One covariate, or a learned combination of several, plays the role of baseline immune infiltration. Immunotherapy alone helps in proportion to infiltration. Adding radiation multiplies the response in low-infiltration patients up to a saturation threshold. A lymphocyte-depletion term subtracts benefit at large simulated field sizes, reflecting the concern raised in the high-grade glioma review. We will vary censoring from 20% to 85%, use both informative and non-informative censoring, sweep confounding strength through the radiotherapy-assignment model, and vary how strongly effect size depends on infiltration. Additive-effect families stay in the testbed as controls.

The third component is the selector. For each simulated cohort we will compute the diagnostics $z$ and evaluate the selection regret of each criterion $j$:

$$R_j = \mathrm{RMSE}\big(\hat\tau_{k_j}\big) - \min_k \mathrm{RMSE}\big(\hat\tau_k\big),$$

where $k_j$ is the estimator that criterion $j$ ranks first, and RMSE is computed against the simulated true effect on held-out patients. A gradient-boosted classifier $g(z)$ is trained to predict the lowest-regret criterion. It is evaluated leave-one-family-out: it is trained on some covariate sources and outcome families and tested on unseen ones, so it cannot simply memorize a simulator. The candidate pool follows SurvHTE-Bench's three families: imputation learners, direct survival models such as Causal Survival Forests and SurvITE, and meta-learners built on DeepSurv and random survival forests.

The scope is deliberately narrow. The treatment is binary (radiation added or not), the context is cohort-level, and the reward is survival only. Continuous dose, spatial-histology routing, and toxicity trade-offs are real gaps, but they would multiply the unknowns before the basic question is settled.

Baselines

The naive reference is factual cross-validation: pick the estimator with the best held-out concordance index or integrated Brier score on observed outcomes. Curth et al. argue this is uncorrelated with effect-estimation error in uncensored data. We expect high regret here as well, and any method that fails to beat it is not worth pursuing. A second simple reference is the global default: always deploy the survival meta-learner on DeepSurv, which SurvHTE-Bench reports as the top overall performer. That paper gives no numeric RMSE in the material I have, so no reported result found on Lacuna for a scalar SurvHTE-Bench error. Its regret will be measured directly.

The strongest existing competitor is a fixed censoring-aware doubly robust criterion applied in every regime. In other words, it is our first component without the selector, run with and without weight truncation. It is the natural censored extension of the criterion Curth et al. found least bad. We will also run Guo & Gao's error-controlled procedure with the same pseudo-outcome plugged in. On uncensored Twins they report an average number of wrong selections of 0.50 against 0.88 for naive selection (Guo & Gao). We will report the same statistic on our censored Twins variants to see how much of that gain survives censoring.

A meta-feature selector in the style of McElfresh et al. predicts the best estimator directly from $z$, with no validation criterion. This tests whether criteria are needed at all. A uniform average of all candidate estimates tests whether selection beats hedging. The key ablation removes the dependence on $z$: an oracle-free "best single criterion on training families" rule. A second ablation drops the estimator-disagreement diagnostic to check whether censoring fraction alone carries the signal.

All methods see identical cohorts, nuisance fits, candidate pools and fold splits. The selector and the meta-feature model get the same training families and hyperparameter budget. We will count the proposal as winning if it lowers mean regret relative to the fixed doubly robust criterion in the above-70%-censoring regime, by a margin outside the across-seed confidence interval, and is not worse below 50% censoring.

Why not just…?

Why not just use the censoring-weighted doubly robust criterion everywhere? This is the real threat. If its variance stays manageable at high censoring, possibly with truncated weights, the selector adds nothing, and that outcome would be a useful negative result in its own right. The regret-versus-censoring curves for this criterion are the first thing we compute. If they stay flat, we report that and drop the selector.

Why not just deploy SurvHTE-Bench's top estimator for everyone? SurvHTE-Bench itself reports no universal winner across its 52 datasets. Priming-shaped multiplicative effects also fall in the regime where On Inductive Biases shows fixed structural choices fail. The global-default baseline measures how much this costs in our testbed.

Why not just fit a Cox model with a radiation-by-immunotherapy interaction after propensity matching, as the clinical studies do? This assumes proportional hazards and a single interaction coefficient. SurvITE reports that Cox models do worse than balanced deep survival models under event-induced covariate shift. The Cox-interaction model enters the candidate pool, so if it is ever the best choice, the selectors are free to pick it.

Why not just average all estimators? Averaging hedges against bad picks but dilutes the estimator that captures the threshold effect. The uniform ensemble is a baseline. If it matches selection, that is reported as the practical recommendation.

Decisive Checks

The central check is the crossover. Plotting each criterion's regret against censoring fraction on held-out outcome families should show the doubly robust and plug-in curves crossing somewhere around 60–75% censoring. The selector should follow the lower envelope of the two. If there is no crossover, the core claim is false. The second check is congeniality under priming. On the multiplicative families, we will compare how often a plug-in criterion built on a random survival forest picks forest-based estimators, relative to the additive families. This tests whether Curth et al.'s bias grows when the true effect is non-additive. The third check is policy relevance. For each selected estimator we will measure the fraction of simulated low-infiltration patients correctly recommended radiation. This translates regret into whether the cold tumors that should benefit actually get flagged. A final sanity check is the Zhang et al. esophageal setting. We will simulate a cohort with their reported structure: about 730 patients, a histology split, and a subgroup-only benefit. We then ask whether criterion-based selection recovers the subgroup effect that pooled analysis misses. This checks plausibility only, since their individual-level data are not available to us.