Choosing Survival Architectures That Hold Up Past the End of Follow-Up

Many neurological treatments only work if given in time. Immunotherapy for autoimmune encephalitis given after about 28 days, or treatment of infantile spasms that responds only after more than 3 months, is followed by chronic epilepsy or relapse far more often. What makes these delays dangerous is that the harm often shows up late, as relapse or entrenched epilepsy months or years afterward. Clinical cohorts, however, are usually followed for only a short time.

When we use automated search to pick a neural survival model (a model that predicts how an event's risk evolves over time from data where many patients leave the study before the event), every candidate is judged only on the events that happened before follow-up ended. We argue that this makes automated architecture search on short follow-up systematically favor models that fit early risk well and extrapolate badly. The models it prefers blur a delayed, threshold-shaped effect of treatment latency into a flat, early-risk summary, and standard ranking metrics cannot see the mistake.

This note proposes to measure that failure and to test one fix. The fix is a "backtest" fitness signal that scores each candidate by how well it forecasts events in the last stretch of the observed window after being trained only on the earlier stretch. The contribution is therefore not "neural architecture search for survival." It is a claim about how model selection should be scored when the quantity that matters, long-term durability after a time-critical treatment, lies beyond the data. We test that claim on long-follow-up survival benchmarks that we truncate on purpose, so that the truth beyond the cutoff is known.

Core Claim

Three findings in the file motivate the question.

First, therapeutic latency produces effects whose hazard ratio changes over time. Deckard et al. found that spasm duration over 3 months before response raised 12-month relapse risk (hazard ratio 3.11). They also had to censor all follow-up at 12 months because longer follow-up broke the Cox proportional-hazards assumption, the assumption that a covariate multiplies risk by the same factor at every time. The late part of the latency effect is exactly what was cut off.

Second, Lillelund et al. show that Harrell's C-index, the usual survival metric, which counts correctly ordered pairs of patients, can report gains while calibration worsens under non-random censoring. They also report that about 72% of recent survival papers use a metric that does not match their censoring assumptions.

Third, AgentNAS is an agentic architecture search in which a language model proposes a seed network and a "slotted" search space that evolutionary search then refines. It reports search overfitting to validation metrics on a genomic task.

Put together, these findings imply that search done in the short-follow-up regime is aimed at the wrong target.

Our claim is that, when a treatment-latency effect is delayed, the architecture chosen by short-window fitness (C-index or even censoring-weighted Brier score) has high regret. Here regret means how much worse the chosen architecture's survival-curve error beyond the cutoff is than that of the best candidate in the same search space.

We further claim that a backtest fitness recovers most of that regret. It works as follows:

  • Train each candidate on data additionally censored at an earlier time $\tau_1$.
  • Score its forecasts over the interval $(\tau_1, \tau]$, where $\tau$ is the end of available follow-up.
  • Refit the chosen candidate on the full window.

Concretely, the fitness of candidate $a$ is

$$F(a) = \frac{1}{\tau - \tau_1}\int_{\tau_1}^{\tau} \mathrm{BS}_{\mathrm{IPCW}}(t;\, a_{\le \tau_1})\, dt,$$

where $a_{\le \tau_1}$ is the candidate fit only on information up to $\tau_1$. $\mathrm{BS}_{\mathrm{IPCW}}(t)$ is the Brier score at time $t$, the squared error of the predicted survival probability, reweighted by the inverse probability of remaining uncensored.

The bet rests on one assumption. How a model extrapolates from $\tau_1$ to $\tau$ must predict how it extrapolates from $\tau$ onward. That assumption can fail, and testing it is the central experiment.

Research Program

The testbed. We build a truncation testbed from the survival benchmarks already used for deep survival models: SUPPORT, METABRIC, GBSG/Rotterdam, UNOS and KKBOX, as used in Survival Kernets and DeepSurv. For each dataset we impose an artificial end of follow-up $\tau$ at several fractions of the observed time range. The events and censoring after $\tau$ are held out as the "future."

Dependent censoring. Administrative truncation alone is independent censoring, which is the easy case. We therefore add dropout that depends on covariates, following the censoring mechanisms Lillelund et al. show break standard metrics.

Semi-synthetic latency outcomes. Because none of these datasets records treatment latency, we add a semi-synthetic arm. We keep the real covariates, add a simulated time-to-treatment variable, and generate outcomes whose hazard, above a patient-specific latency threshold, shifts mass into a late relapse phase.

The parameters are calibrated to the published cohorts:

  • a roughly 21% relapse rate and a hazard ratio near 3 for latency beyond a threshold, from the infantile spasms cohort;
  • an odds ratio near 6.4 for immunotherapy after 28 days, from the encephalitis cohort of Zhong et al.;
  • cohort sizes in the low hundreds, matching those cohorts.

The threshold depends on covariates, which reflects the "individualized window" open problem. The simulation follows the style of DeepSurv's non-linear risk simulation but moves the latency effect to late times.

The search space. We use a slotted search space in the manner of AgentNAS, with three slots:

  • Latency encoding: raw input, spline basis, monotone network, or a learned soft-threshold gate.
  • Hazard head: Cox-type proportional head as in DeepSurv, discrete-time multi-horizon head, or time-varying-coefficient head.
  • Connectivity: the trunk's shortcut pattern, including learned shortcut weighting in the style of ANCRe.

Classical models are candidates in the same pool: Cox with spline latency and time-varying coefficients, and random survival forests. This makes the question one of selection, not of neural versus classical.

To keep cost bounded we warm-start neural candidates with the tree-ensemble initialization (TUNA) from Survival Kernets. Every fitness criterion gets the same number of candidate evaluations.

Regret measurement. For each criterion we measure selection regret against the oracle best candidate. In the semi-synthetic arm we also measure how accurately the selected model recovers each patient's latency threshold.

Baselines

Naive reference point. A linear Cox model with latency entered as the population cutoff indicator, as the clinical papers do (for example, over versus under 3 months), fit on data up to $\tau$ and extrapolated using its baseline hazard. Anything that cannot beat this on survival error beyond $\tau$ has no clinical case.

Strongest classical competitor. Cox regression with a spline in latency and a latency-by-log-time interaction. This directly models the non-proportional effect that forced Deckard et al. to truncate. It is tuned by cross-validated partial likelihood with the same number of configurations as each search run.

Strongest hand-designed neural reference. DeepSurv, a feed-forward network inside a Cox objective. The file reports its METABRIC C-index as 0.867, against 0.816 for linear Cox and 0.619 for random survival forests (DeepSurv). Survival Kernets is the second neural reference, with its time-dependent concordance described in the file as comparable to DeepHit and DeepSurv. The file gives no numerical results on Lacuna for Survival Kernets, or for DeepSurv on SUPPORT, GBSG/Rotterdam, UNOS, WHAS or KKBOX. We rerun both on our splits instead of quoting numbers.

Strongest search baseline. AgentNAS-style evolutionary search over the same slotted space, guided by C-index. Its reported results are 11 of 17 tasks won and 13.06% average error on the unseen benchmarks, against 15.04% for the best conventional search (AgentNAS). Those numbers are for non-survival tasks and serve only as evidence that the search engine is competent.

Key ablations. Two ablations isolate the proposal's ingredient: the same search with integrated censoring-weighted Brier over $(0, \tau]$ as fitness, and the same search with random-candidate selection.

All methods share splits, truncation points, candidate budget and warm-starting. Beating the baselines means lower integrated censoring-weighted Brier and lower absolute survival-curve error beyond $\tau$ than the best of them, significant across repeated truncation seeds. In the semi-synthetic arm it additionally means lower error in the recovered latency threshold.

Why not just…?

Why not just fit a Cox model with a flexible time-varying latency effect and skip search entirely? This is the most serious threat, especially at cohort sizes near 200. If the spline-Cox baseline matches backtest-selected search on error beyond $\tau$, the lesson is that selection criteria matter less than model class, and we will report it that way. Because spline-Cox sits inside the candidate pool, the backtest can also select it. What the experiments will show is whether short-window criteria pass it over in favor of models that fit early data better and extrapolate worse.

Why not just evaluate with censoring-aware metrics such as the one Lillelund et al. recommend? Reweighting for censoring corrects bias only for times where some patients remain observed. It says nothing about how a model behaves after $\tau$, which is where delayed latency effects live. The "Brier over $(0,\tau]$" ablation tests this directly. If it matches the backtest, our extra machinery is unnecessary.

Why not just report predictions restricted to the observed horizon and refuse to extrapolate? For questions answered within the window, that is the honest choice, and our results should not be read as licensing extrapolation. But whether to treat inside or outside a window depends on long-term durability, and in practice clinicians extrapolate implicitly. Our aim is to measure which selection rule makes that implicit extrapolation least wrong, and to flag when none does.

Decisive Checks

The central check is rank agreement. For each dataset and truncation, we compute Kendall's tau between the candidate rankings produced by each fitness criterion on data up to $\tau$ and the oracle ranking computed on the held-out future. If backtest ranking does not agree with the oracle better than short-window Brier does, the assumption behind the proposal fails. We will also report how that agreement decays as $\tau_1$ moves away from $\tau$, which tells practitioners how much follow-up the backtest needs.

Where the effect should appear. In the semi-synthetic latency arm, the regret gap between C-index-guided and backtest-guided search should grow as the latency effect is placed later relative to $\tau$. It should vanish when the effect is proportional. Seeing both is what ties the gap to therapeutic-window dynamics specifically.

Small cohorts and dependent dropout. We repeat everything at subsampled cohort sizes of 100–300, the size of the published cohorts, and under covariate-dependent dropout. If the gap disappears at clinical sample sizes, search of any kind is not worth doing there, and the recommendation reduces to the spline-Cox baseline.