Calibrating Ordinal Disease-Stage Forecasts to the Patient Population, Not the Clinic

Many clinical risk tools forecast a patient's future disease stage. Examples are the five stages of chronic kidney disease or a ten-stage Alzheimer's progression scale. These models are trained on routine records, and in routine records sicker patients are measured more often. A model trained this way sees deteriorating patients many times and stable patients rarely. Its predicted stage probabilities can look well calibrated, meaning predicted frequencies match observed ones, on a held-out set drawn from the same clinic. Yet they can be badly miscalibrated for the population a clinician actually cares about, which is everyone under follow-up, measured or not. The bias also has a characteristic shape on an ordered scale. Forecasts should spend too much probability on transitional, high-acuity stages and too little on long stays in stable early stages.

We propose a post-hoc procedure, plus a controlled study, that makes ordered stage forecasts calibrated with respect to the population. The procedure has two parts. First, it reweights each observed visit by how unlikely that visit was. This uses an inverse visit-intensity weight that is itself stabilized by isotonic calibration, a monotone curve-fitting step. Second, it applies a jointly monotone recalibration to the model's cumulative stage probabilities $P(Y \le k)$, using those weights. The recalibration keeps the stage ordering intact, so probabilities never cross and prediction sets never fragment. The study then asks something the literature has not tested: once evaluation is done honestly with respect to the population, does training-time reweighting add anything beyond this cheap post-hoc fix?

Core Claim

Calibration of ordinal stage forecasts on informatively sampled longitudinal data has to be defined against the right distribution. Three conditions together achieve it: weights correct for visit timing, the weights themselves are calibrated so they do not explode during long gaps, and the recalibration map respects the order of the stages. Each piece exists separately, but no existing method combines them.

Inverse intensity weighting for irregular visits comes from Accounting For Informative Sampling When Learning to Forecast Treatment Outcomes Over Time (TESAR-CDE). That work treats visit timing as a point process and down-weights periods of frequent visits. It handles only continuous outcomes, never assesses calibration, and applies no stabilization to the estimated intensities.

Isotonic calibration of inverse weights comes from Stabilized Inverse Probability Weighting via Isotonic Calibration (IC-IPW). That paper shows that calibrating a propensity score does not calibrate its inverse. It fixes this with a monotone remapping plus adaptive truncation, but only for static binary treatments.

Threshold-based ordinal heads come from A neural network approach to ordinal regression. These are known to suffer threshold crossing, and none has been evaluated under informative sampling.

Our concrete claim is twofold. Calibration of the weights is what makes reweighting usable at the rare-visit tail. Monotone, order-preserving recalibration is what keeps the ordinal structure intact after reweighting. We also predict a measurable failure without the method: unweighted models, even after ordinary recalibration, will show stage-specific cumulative calibration error concentrated at the early-stage thresholds.

Research Program

Let $x_i$ be the history available at visit $i$ and $y_i \in \{1,\dots,K\}$ the stage observed at a forecast horizon. The base model, which can be any ordinal forecaster, outputs cumulative probabilities $\hat F_k(x_i) \approx P(Y \le k \mid x_i)$.

Step 1: estimate visit intensity. We first fit a visit-intensity model $\hat\lambda(t \mid \text{history})$. This can be the intensity head of TESAR-CDE or a simpler Poisson model.

Step 2: calibrate the intensity, and therefore the weights. Following the logic of IC-IPW, we calibrate the intensity before inverting it. We fit an isotonic Poisson regression of observed visit counts on $\hat\lambda$, with exposure time as the offset. This produces a monotone remapping $\tilde\lambda$ for which the realized visit rate matches the predicted rate within every level of $\hat\lambda$. We then set $w_i = 1/\tilde\lambda(t_i)$ and apply adaptive truncation at the boundary blocks.

Correctness here means the weights balance, in the sense that $E[w\,dN(t) \mid w] \approx dt$, where $dN(t)$ is the visit counting process. This target is the continuous-time analogue of the balance-based calibration used in IC-IPW. Extending IC-IPW to intensities is one deliverable.

Step 3: fit the order-preserving recalibration. We fit maps $g_1,\dots,g_{K-1}$ by minimizing a weighted ranked probability score, which is the sum of squared errors over the cumulative indicators:

$$\min_{g}\ \sum_i w_i \sum_{k=1}^{K-1}\big(\mathbf{1}[y_i \le k] - g_k(\hat F_k(x_i))\big)^2,$$

The minimization is subject to two constraints. Each $g_k$ must be non-decreasing, and $g_1(\cdot) \le \dots \le g_{K-1}(\cdot)$ must hold pointwise.

This is a stacked isotonic problem solvable by alternating pool-adjacent-violators. It uses the multi-horizon squared-error idea that Estimating Calibrated Individualized Survival Curves with Deep Learning applied to survival curves, but here across ordinal thresholds. The crossing constraint directly removes the threshold-inconsistency failure of cumulative models.

Step 4: test training-time weighting. Separately, we train ordinal threshold heads with the same calibrated weights inside the loss. This tests whether reweighting during training changes the learned representation in ways post-hoc recalibration cannot.

The no-confounder assumption behind all of this, that visits depend only on recorded history (called Sampling At Random), is the method's main limitation. We treat it as something to probe, not assume. A simulation arm therefore lets visits be triggered by unrecorded latent shocks.

Baselines

Naive reference. The do-nothing baseline is an unweighted cumulative-threshold ordinal network trained and evaluated on the observed visits. It is the honest description of what a clinic would deploy.

Strongest calibration competitor. The strongest competitor is the same network followed by standard post-hoc recalibration fit on an unweighted, informatively sampled validation set. We will run both the per-class isotonic version used in Multidimensional Machine Learning Personalized Prognostic Model in an Early Invasive Breast Cancer Population-Based Cohort in China and temperature scaling on a softmax head. This is the strongest competitor because it is what practitioners actually do, and it will look perfectly calibrated under the usual evaluation.

Strongest in-training calibration method. We will also run an in-training differentiable calibration penalty, adapting X-CAL: Explicit Calibration for Survival Analysis to cumulative ordinal targets. X-CAL reports reducing D-calibration error "often by an order of magnitude" on MIMIC-III. Like Kamran and Wiens, it assumes uninformative observation.

Weighting competitors. We will run TESAR-CDE-style inverse intensity weights without calibration, and the same weights clipped at the 95th and 99th percentiles.

Ablations. The ablations that isolate our ingredients are three: calibrated weights with per-threshold recalibration but no crossing constraint; the crossing-constrained recalibration with uncalibrated weights; and a unimodal Binomial head from Unimodal Probability Distributions for Deep Ordinal Classification under the same weighting.

Fair-comparison protocol. All methods share the same backbone, the same train, calibration, and test splits, and the same hyperparameter search budget. Weighting and recalibration are fit only on the calibration split.

What counts as beating a baseline. The method wins if, on test data evaluated against the population ground truth, the following hold together:

  • population-weighted ranked probability score and per-threshold calibration error both improve over the standard-recalibration baseline, with paired bootstrap intervals excluding zero;
  • discrimination, measured as ordinal concordance, is not lost by more than 0.01;
  • crossing violations are zero.

No reported result was found on Lacuna for ordinal stage calibration under informative sampling on any benchmark in the file. The closest reported calibration numbers come from Kamran and Wiens on a different task: DDC 0.007 versus 0.025 for DRSA on NACD, with a C-index of about 0.74.

Why not just…?

Why not just recalibrate on held-out data as usual? The held-out data come from the same visit process. Ordinary recalibration therefore calibrates to the clinic visit distribution, which is exactly the gap described in the file's analysis of survival calibration under informative sampling. The decisive experiment compares visit-distribution calibration error against population calibration error for this baseline as the informativeness parameter grows. We expect the first to stay near zero while the second rises.

Why not just add the inverse weights to post-hoc recalibration and skip everything else? This is the real threat, and it is largely our method. The open question is which additions matter. If uncalibrated or clipped weights perform as well as calibrated ones, then the IC-IPW extension is unnecessary and we will say so. The ablation grid decides this, especially in the rare-visit tail, where Lee et al.'s argument predicts that uncalibrated inverses fail.

Why not just clip the weights? Lee et al. report that truncation trades variance for uncontrolled bias, and that IC-IPW improved bias, RMSE, and coverage on the limited-overlap ACIC-2017 processes. They give no numerical tables, so we test the claim directly by sweeping clip levels against calibrated weights.

Why not just regress the stage number with mean squared error, or use softmax with temperature scaling? Neither produces order-consistent cumulative probabilities. Softmax heads give multi-modal posteriors and non-contiguous prediction sets, as reported by Beckham et al. We include both and report prediction-set contiguity alongside calibration error.

Proposed Experiments

Arm 1: simulation with full ground truth. We use the lung-cancer tumor-growth simulator from Vanderschueren et al. and discretize tumor volume into $K=5$ ordinal stages. Its informativeness parameter $\gamma$ sweeps visit timing from completely random to fully history-dependent. Because the full latent trajectory is known, population calibration can be computed exactly. We add a variant in which visits are triggered by an unrecorded shock, to measure how far each method degrades once the core assumption fails.

Arm 2: semi-synthetic real data. ADNI visits are largely protocol-scheduled, so stages at scheduled visits approximate a population reference. Stages are EuroPOND event-based stages 0–9, following Independent validation and outlier analysis of EuroPOND Alzheimer's disease staging model using ADNI and real-world clinical data, with diagnostic categories as a fallback. We thin these visits with acuity-dependent dropout to imitate memory-clinic attendance. Models see only the thinned visits and are scored on the full schedule.

Arm 3: real-world observational data. The Duke CKD cohort from Scalable Joint Modeling of Longitudinal and Point Process Data for Disease Trajectory Prediction and Improving Management of Chronic Kidney Disease reports 23 eGFR readings per patient on average. CKD stages follow deterministically from eGFR. This arm offers no ground truth, so it serves only as a consistency check, testing whether weighted and unweighted calibration curves diverge in the predicted direction. Access is not guaranteed, and the study does not depend on it.

Decisive checks. Three results settle the claim. First, in Arm 1, the standard-recalibration baseline's population calibration error grows with $\gamma$ and concentrates at early-stage thresholds, while our method's stays flat. Second, calibrated weights beat clipped weights in the high-$\gamma$, long-gap regime. Third, training-time weighting either does or does not improve over post-hoc weighting at matched discrimination. Each outcome is informative: a null on the third would mean the cheap post-hoc procedure is the whole answer.