When Should a Deployed Model Be Retrained? Survival Analysis of Model Decay Under a Carbon Budget

Models in production get worse as the world changes. This is called temporal distribution shift. The standard fix is to keep watching the model and retrain it when it degrades. Studies of this problem treat retraining as free. Carbon accounting work shows it is not, and the cost grows as the models being maintained get larger. We propose to treat a deployed model's useful life as a time-to-event outcome. The "event" is the moment the model's ranking ability falls below an acceptable level. We would fit a continuous-time survival model to this outcome. A survival model predicts how long until an event happens and correctly handles cases where the event has not yet been observed. With it, we would choose when to recalibrate cheaply, when to retrain fully, and when to do nothing. The choice weighs the predicted loss in usefulness against the actual carbon cost of each action.

The key observation is that the data on model lifetimes is censored by construction. Every time a team retrains early, it never learns when the old model would have failed. Labels for time-to-event tasks also arrive late. A simple performance monitor with a threshold therefore sees decay too late and only partially. Survival analysis exists to handle exactly this kind of incomplete observation. Carbon accounting tells us which costs belong in a decision made today and which do not.

Core Claim

The drift literature agrees that reactive maintenance beats building robustness in advance. In clinical prediction on MIMIC-IV, domain generalization and adaptation methods did not beat plain training on all available data (Guo et al., DG/UDA under temporal shift). Across five longitudinal benchmarks, standard training on pooled data proved very hard to beat, and performance fell steadily as the gap between training and test years grew (Wild-Time). A systematic review further found that drift damages calibration well before it damages discrimination (Guo et al., systematic review). Calibration is whether predicted probabilities are honest. Discrimination is whether the model still ranks cases correctly. Cheap recalibration fixes the first problem but not the second.

Taken together, these findings make maintenance a decision problem with two clocks: time until calibration fails and time until discrimination fails. Each failure has a remedy with very different costs. Our claim is that a hazard model of model decay can schedule these two remedies better than a fixed calendar or a threshold alarm. The hazard model would be trained on many simulated deployment histories. It would take unlabeled drift statistics and time since training as inputs. It would treat early retraining as right-censoring, meaning the failure time is known only to exceed the retraining date. We expect the improvement to be largest where labels are delayed and where retraining is expensive.

The carbon side of the decision follows the Sunk Carbon Fallacy. That work shows that charging embodied emissions to individual run-time decisions pushes work onto inefficient hardware and raises total emissions. Embodied emissions are those released when the hardware was manufactured. Accordingly, the per-decision retraining cost in our policy is marginal operational carbon only: energy times grid carbon intensity at the time and place of the run. Embodied carbon enters at a different level. The hazard forecasts for a whole portfolio of models predict peak retraining demand. That demand determines how much accelerator capacity must be bought, and capacity purchases are where embodied carbon properly belongs.

Research Program

The first component is a meta-dataset of model lifetimes. We build it by training models on rolling windows that end at many different calendar dates, then evaluating each model on every later period. This yields thousands of decay curves per dataset. We use the Wild-Time datasets with sub-year timestamps (arXiv, HuffPost) and the year-resolution ones (Yearbook, FMoW-Time). For each curve we record two event times: when calibration error crosses a tolerance and when discrimination drops a fixed amount below its value at deployment. MIMIC-IV publishes dates only in 3-year anonymized groups. We therefore treat its events as interval-censored, meaning we know only which window the failure fell in. Continuous-time models handle this naturally through differences of the cumulative distribution function. Discrete-bin models cannot represent it without imputing a time within the interval.

The second component is the decay hazard model. It is a continuous-time survival model whose inputs include the deployment age, label-free drift measures computed on incoming inputs (shifts in the embedding distribution and in the distribution of predicted scores), training-window length, and model family. We use the Asymmetric Laplace parameterization of Sheng and Henao as the main model. That method uses a neural encoder to predict the location, scale and skew of an asymmetric Laplace distribution, giving closed-form survival and hazard curves. It beat DeepHit in 73% and LogNorm in 60% of 189 settings. We adopt it because a scheduler needs exact hazards and quantiles at arbitrary times. The same paper reports trouble when events cluster near time zero. Abrupt pipeline changes create exactly that pattern. We therefore add a point mass at zero, a probability of immediate failure, as an extension, and test whether it is needed.

The third component is the maintenance policy. At each decision time the policy compares expected utility lost over a planning horizon under three actions: keep the model, recalibrate it, or retrain it. Retraining is charged its marginal operational carbon $c_{\text{op}}(t)$. That cost is drawn from a time-varying distribution in the style of PCAM, which models carbon uncertainty with kernel density estimates over hourly grid data. A single exchange rate $\beta$, in utility units per kilogram of CO₂-equivalent, traces out a Pareto front. Because the hazard model is continuous, the policy can also move a retraining run a few days earlier or later to land on a cleaner grid hour. A daily or monthly binned model cannot express this trade.

The fourth component, a smaller study, asks whether the asymmetry between calibration decay and discrimination decay holds for survival models themselves. We redefine MIMIC-IV mortality as time-to-death censored at discharge and deploy Asymmetric Laplace models across the anonymized year groups. This addresses a gap identified in the research file: existing drift benchmarks reduce time-to-event outcomes to binary labels. If survival models lose discrimination as fast as calibration, then recalibration stops being a useful cheap action in the clinical setting, and the policy simplifies.

Baselines

The naive reference is to deploy once and never retrain. It sets the floor and shows how much decay exists to be recovered. Wild-Time reports mortality AUC falling from 0.90 to 0.72 on MIMIC-IV and Yearbook accuracy falling from about 98% to about 79% from in-distribution to future data. Guo et al. report a maximum AUROC drop of 0.090 for sepsis prediction.

The strongest existing practice is to retrain on a fixed calendar using all data so far. This is the approach Guo et al. found best (training on 2008–2016) and Wild-Time found hard to beat. We run it at several cadences under the same model architectures, training compute per run, and label-delay model as our policy. We report its full utility–carbon curve rather than a single point. No reported result found on Lacuna for any carbon-aware or decay-forecasting retraining policy, so all policy-level comparisons are new measurements.

The strongest reactive competitor is a threshold monitor. It retrains when performance measured on labels arriving with realistic delay drops by a set amount. It also recalibrates with Platt scaling when calibration error crosses a tolerance. Its thresholds get the same tuning budget, on the same validation periods, that we spend on $\beta$ and the decision horizon. A second reactive competitor triggers on the unlabeled drift statistics alone, with no survival model. It tests whether the hazard model adds anything beyond its inputs.

We run three ablations. Replacing the continuous hazard with a discrete-time hazard on monthly bins isolates the effect of continuity. Ignoring censoring by fitting only complete lifetimes isolates the effect of censoring. Charging amortized embodied carbon per retraining run tests whether the Sunk Carbon Fallacy appears at the decision level, which we predict it will. We count the proposal as beating the baselines if, at equal cumulative retraining carbon, it achieves higher area under the deployment-performance curve than the best-tuned calendar and threshold policies on held-out datasets. Alternatively, it beats them if it matches their performance with substantially less carbon.

Why not just…?

Why not just retrain on a fixed schedule with plain training on all data? For small models this may be nearly optimal, because retraining is close to free. We will report this as a result, not hide it. The case against it rests on two findings. Guo et al. show that much early decay is calibration, which recalibration fixes at negligible cost. Wild-Time shows decay rates differ greatly across datasets, so any single cadence is too frequent for some and too slow for others. Our experiments test whether those differences are large enough, and retraining expensive enough, for a forecast-driven schedule to win.

Why not just monitor performance and retrain when it drops? This is the real threat. With fast, complete labels it may match our policy. The difference should appear only under label delay and censoring, which are normal for time-to-event outcomes. Our experiment therefore varies label delay explicitly. If the threshold monitor matches the hazard policy even at long delays, the survival component is unnecessary, and we will say so.

Why not just use a discrete-time hazard model? Discrete models are simpler. The case for continuous time here is narrow and testable. It has two parts: interval-censored MIMIC-IV events, and shifting retraining runs to cleaner grid hours within a bin. The monthly-bin ablation measures both directly.

Proposed Experiments

The first decisive check is whether the decay hazard model generalizes. We train it on lifetimes from three Wild-Time datasets and test it on the held-out fourth and on MIMIC-IV. We measure concordance index, integrated Brier score and distribution calibration. A model that only memorizes per-dataset decay rates fails this check.

The second check is the policy comparison. We trace Pareto fronts of cumulative deployment performance against retraining carbon for every baseline. We do this at three model scales, from logistic regression to fine-tuned transformers, so that the regime where carbon matters is visible. Carbon costs come from hourly grid traces of the kind CarbonClipper uses. CarbonClipper optimizes where and when batch jobs run but never asks whether a job should run at all. The third check is the capacity-planning study. We compare the accelerator fleet size implied by the hazard forecasts with the size implied by calendar retraining. We then convert the difference into embodied carbon using peripheral-inclusive server models from SCARIF. The fourth check is the MIMIC-IV survival-drift study. It reports separately how fast calibration and discrimination decay under the anonymized year groups. It also states whether recalibration remains a useful cheap action for continuous-time clinical models.