When Short Follow-Up Picks the Wrong Training Length

Clinical risk models are almost always trained on cohorts whose follow-up was cut short. Patients are watched for months or a few years, and anyone still event-free at the end is "censored": we know only that they survived at least that long. These models are also usually trained for a heuristic number of epochs, with the stopping point chosen by a validation score computed on the same truncated window. That is two truncations stacked on top of each other. This proposal argues they are not independent. When follow-up is short, the validation signal used to decide how long to train can only see early events. It may therefore reward models that fit fast-acting risk factors and penalize models that, with more training, would rank patients better for events that happen later. We propose to measure this effect directly and to test a cheap correction.

The measurement uses cohorts that actually have long follow-up, such as METABRIC, SUPPORT and SEER. We cut their follow-up artificially at a range of horizons, train deep and classical survival models over a grid of training lengths, and score each model against the outcomes that truncation hid. The correction is a "truncation ladder." Within whatever follow-up a real study has, we truncate further (to half, then a quarter of the window) and ask which training configurations, trained on the shorter window, best predict events in the next stretch of time. The configurations that win those look-ahead tests are the ones we trust for the future beyond the study's end.

Core Claim

Our claim is that, under heavy censoring from short follow-up, choosing training duration and architecture by in-window validation produces systematic regret on long-horizon discrimination and calibration. The regret grows with the censoring fraction. It also produces rankings between methods that invert across follow-up horizons, in the same way that data-quality rankings inverted across epoch budgets in Call for Rigor in Reporting Quality of Instruction Tuning Data. That paper showed that curated LIMA and length-filtered Alpaca-Longest swap places depending on learning-rate schedule and on 3 versus 15 epochs. We expect the analogous inversion here, with follow-up horizon as a second axis crossed with training length. The secondary claim is that a look-ahead selection score, computed only from data inside the study window, recovers most of this regret. If it does not, the result tells us that the horizon effect is not self-similar and cannot be corrected from within-study data.

Why This Gap Exists

Two literatures motivate the question and neither answers it. On the optimization side, Revisiting Few-sample BERT Fine-tuning showed that instability on small datasets was largely an artifact of stopping too early combined with an optimizer flaw. Training up to 20x longer with bias-corrected Adam cut the RTE failure rate from 48% to near zero and cut the random restarts needed for top performance from 50+ to 5–10. ProRL makes a stronger version of the same point: behaviours absent at a few hundred reinforcement-learning steps appear after 2,000+. Both papers establish that stopping early measures speed of adaptation rather than capacity. Neither involves censored outcomes, where the evaluation signal is itself truncated.

On the survival side, several neural models handle censoring in the loss but train to fixed or early-stopped budgets. DeepSurv is a neural network trained with the Cox partial likelihood. DeepHit predicts a discrete distribution over event times and is trained with a likelihood term plus a pairwise ranking term. SurvTRACE is a Transformer with propensity-weighted hazard loss and auxiliary heads. None of the three studies how the training schedule interacts with how much follow-up the data contain. SurvHTE-Bench comes closest. It varies censoring up to and beyond 70% across 53 treatment-effect estimators and finds that survival-aware meta-learners built on DeepSurv dominate under heavy censoring. It trains every estimator to fixed stopping criteria, however, and never scores against outcomes beyond the censoring point. We borrow its design of controlled censoring and its codebase, and add the missing axis.

The clinical papers in Direction 2 give the stakes and constrain what the analysis can assume. In Yang et al., an atrial fibrillation recurrence model reached an AUC of 0.814 at 3–6 months while explicitly missing later recurrences. Sohal et al. found a 6-month imaging marker with an odds ratio of 5.67 for reverse remodeling, but concede it says nothing about mortality. Both show that good in-window performance does not certify out-of-window performance. This is the phenomenon we want to quantify for the training loop itself.

Research Program

The first component is a controlled truncation study. For each long-follow-up cohort, we set administrative horizons $\tau$ at quantiles of the observed event-time distribution, chosen so that the censoring fraction is roughly 30%, 50%, 70% and 85%. All patients still at risk at $\tau$ are censored.

For each horizon we train a grid of models:

  • Models: Cox proportional hazards, Random Survival Forests, DeepSurv, DeepHit, and SurvTRACE.
  • Training length (neural models only): from 0.25x to 20x a standard budget.
  • Learning rates: two per model, with checkpoints saved throughout training.

Every checkpoint is scored twice:

  • In-window score: the censoring-weighted concordance at $\tau$. This is what a practitioner would see.
  • Long-horizon score: concordance and integrated Brier score (a squared-error measure of predicted survival probabilities over time) at $2\tau$ and $4\tau$, computed on held-out patients with their untruncated follow-up.

The primary quantity is selection regret. This is the gap in long-horizon score between the configuration chosen by the in-window score and the configuration that is actually best at the long horizon. We also report Spearman correlation between the two scores across configurations, and whether method rankings invert between horizons.

The second component probes mechanism. DeepHit combines a likelihood term with a ranking term, and the number of comparable patient pairs available to the ranking term shrinks as censoring grows. We log the two terms separately to test whether the ranking term converges later than the likelihood term. If it does, early stopping on total loss would stop before discrimination has settled. We also measure how the optimal training length scales with the number of observed events rather than the number of patients. The BERT result suggests that effective sample size governs how long training must run, and under censoring the effective sample size is the event count.

The third component is the truncation ladder. A study with window $\tau$ builds nested sub-windows $\tau_k = \tau / 2^k$. For each configuration $c$, we train on the training split truncated at $\tau_k$ and evaluate on a validation split truncated at $\tau_{k-1}$. This defines a look-ahead score:

$$S(c) = \frac{1}{K}\sum_{k=1}^{K} C_{\tau_{k-1}}\big(f_c^{(\tau_k)}\big)$$

Here:

  • $f_c^{(\tau_k)}$ is the model with configuration $c$ trained on data censored at $\tau_k$.
  • $C_{\tau_{k-1}}$ is the censoring-weighted concordance on validation patients observed out to $\tau_{k-1}$.
  • $K$ is the number of ladder rungs, typically 2 or 3.

We select the configuration $c$ that maximizes $S(c)$ and retrain it on the full window $\tau$. The bet is that "how well does this configuration generalize to the next doubling of time" is roughly stable across scales. That is a falsifiable assumption, and the study tests it directly. The models are small tabular networks, so the full grid costs little. Here the compute barrier that limits ProRL-style studies (about 16,000 H100 hours for a 1.5B model) does not apply.

Baselines

The naive reference point is standard practice. Each neural model is trained with early stopping on in-window censoring-weighted concordance, following DeepSurv's own procedure, with a fixed patience and the default budget. Every other method must reduce this baseline's selection regret to count as an improvement. It runs on identical splits and truncations, and the stopping rule is its only choice.

The strongest existing reference on discrimination is DeepSurv against classical models. On METABRIC with full follow-up, DeepSurv reports a C-index of 0.867, against 0.816 for linear Cox and 0.619 for Random Survival Forests. On nonlinear synthetic data, DeepHit reports a time-dependent concordance of 0.755, against 0.578 for cause-specific Cox. SurvTRACE reports only relative gains of 2.1–1.7% on METABRIC and 1.5–1.9% on SUPPORT. We found no reported result on Lacuna for any of these models evaluated beyond an artificially truncated follow-up horizon. We will therefore first reproduce the full-follow-up METABRIC numbers as a sanity check before introducing truncation. All models get the same hyperparameter search budget, measured in configurations tried, for every selection rule.

The strongest competing selection rule is a fixed long budget. Each neural model trains for a large fixed number of epochs with weight decay, and the final checkpoint is kept. This operationalizes the BERT lesson of "just train long." The ladder must beat it on long-horizon regret at censoring of 70% or more to justify its extra cost.

We also include an arm that applies EXPO to DeepSurv. In Weak-to-Strong Extrapolation Expedites Alignment, EXPO extrapolates weights along the line from an early to a later checkpoint, with a scale factor $\alpha$ that sets how far to extrapolate. That paper matched full alignment training while training on 20% of the data, across 12 models. Here, $\alpha$ is tuned in-window.

Two ablations isolate the ladder's key ingredient. The first uses the look-ahead score with $K = 1$, which is a single sub-window. The second applies the full ladder with training length held fixed and only architecture and learning rate varying. This tests whether the duration axis matters at all.

A win requires two things. First, the ladder must cut mean selection regret at $2\tau$ by a margin exceeding the across-seed standard deviation, on at least two of the three real cohorts at 70%+ censoring. Second, it must not worsen long-horizon integrated Brier score.

Why not just…?

Why not just use Cox or random survival forests, which have no training-length problem? Both are in the grid. DeepSurv's reported full-follow-up advantage (0.867 against 0.816) says nothing about behaviour under truncation. It is plausible that, at 85% censoring, linear Cox has the lowest long-horizon regret, because it cannot fit early-hazard quirks. If so, we will report that deep models should not be used at that censoring level without look-ahead selection. That would be a useful finding, not a failure of the study.

Why not just use censoring-weighted validation, which is the standard statistical fix? Inverse-probability-of-censoring weighting removes bias in the concordance estimate only for times before $\tau$. Beyond $\tau$ there is no observed information, so no weighting can recover it. The naive baseline already uses this weighting. Any regret it shows is precisely the part that weighting cannot fix.

Why not just train long, as few-sample BERT fine-tuning suggests? This is the real threat. Revisiting Few-sample BERT Fine-tuning found that longer training made specialized fixes redundant, and the same could happen here. The fixed-long-budget arm settles the question. If its regret matches the ladder's across cohorts and censoring levels, we will drop the ladder and conclude with the simpler recommendation. We would still be the first to have measured it against hidden long-horizon outcomes.

Why not extrapolate weights with EXPO to simulate more training? EXPO extends optimization time, not observation time. Weights trained on truncated data contain no late-event information to extrapolate. Its authors also note that excessive $\alpha$ causes collapse. The EXPO arm will show whether it adds anything beyond the fixed-long-budget arm. We expect it to track that arm or to degrade calibration.

Decisive Checks

The first check comes before any method work. If in-window and long-horizon scores are rank-correlated above roughly 0.9 across configurations at every censoring level, the core claim is false. In that case the study reduces to a short negative result.

The second check tests self-similarity on the synthetic survival scenarios from SurvHTE-Bench, where true hazards are known. We plant a covariate whose effect appears only after $\tau$, and a second covariate whose effect appears between $\tau/2$ and $\tau$. The ladder should detect the second covariate but, by construction, cannot detect the first. Measuring how much regret each planted effect leaves bounds what any within-study correction can achieve.

The third check is external validation of the whole pipeline on SEER. Its size (470,000+ patients) allows truncation at short horizons while keeping thousands of events. The ladder's selections, made at a short $\tau$, are then scored against multi-year cause-specific mortality.