When two radiologists outline the same multiple sclerosis lesion on the same scan, their masks differ. When one radiologist outlines it on a scan taken a year later, the masks also differ, but now for two reasons: the lesion may really have grown or shrunk, and the radiologist traces boundaries a little differently every time. Current methods treat these two sources of difference separately. Longitudinal segmentation models assume a single correct mask per scan and penalize any change between visits. Models of human disagreement assume every scan is an isolated snapshot. Neither can answer the question clinicians actually ask at follow-up: is this change real, or is it contouring noise?
We propose a model in which the expected real change between two scans is a learned function of the physical time between them, and the expected contouring disagreement is a property of each rater that does not depend on that interval. The interval matters because it is the only lever that identifies the split. Scans read days apart, or the same scan read by two raters, can differ only through rater noise. Scans read years apart differ through both. Fitting a continuous encoding of elapsed time, meaning a learned map from a scalar interval to a vector the network can condition on, lets the model estimate how much disagreement to attribute to biology at every interval, with an intercept at zero elapsed time that equals measured inter-rater variability.
Core Claim
The amount of mask disagreement between two readings of the same patient, expressed as a function of elapsed time $\Delta t$, has a nonzero intercept set by rater noise and a slope set by disease activity. A segmentation model that explicitly factors a time-invariant per-rater noise model from a $\Delta t$-conditioned biological change model will do three things. It will call new and enlarging lesions more reliably than models that either force temporal consistency or ignore time. It will produce rater-specific masks that are temporally coherent without being over-smoothed. It will extrapolate sensibly to intervals it never saw.
What is new is the decomposition and the identification argument, not the time encoding itself. Continuous time encodings already exist: sinusoidal ones in Patra et al. and Wald et al., learned projections in DuETT and TIM, and months-since-baseline in LTSA. Per-rater noise models also exist: confusion matrices in Zhang et al. and Gaussian rater latents in Pionono. We supply the missing link that makes the two jointly identifiable.
Relation to Prior Work
Two longitudinal segmentation papers motivate the gap. Denner et al. concatenate a baseline and a follow-up MS scan and add an auxiliary registration decoder. The model never learns how far apart the scans are, and it is supervised by a single rater. Local Spatiotemporal SSL adds a consistency loss that penalizes segmentation differences between adjacent visits regardless of interval. This penalty is exactly the design our proposal argues against: it treats real change as an error at long intervals and fails to separate rater noise at short ones.
A family of cross-sectional disagreement models supplies our noise component. These include Zhang et al., Pionono, the Latent Doctor Model, OMBA-UNet, MoSE, and MRNet. MRNet's recurrent module runs over an ordering of raters, not over time. We reuse the image-conditioned confusion matrix of Zhang et al. almost unchanged and add what none of them has: a second, time-indexed observation of the same anatomy.
LTSA supplies evidence that elapsed-months encoding beats visit indices on irregular clinical schedules. It reports a 2-year AMD C-index of 0.923 versus 0.886 for single-image baselines. That result is for survival prediction, not masks, so we treat it as motivation rather than proof for segmentation.
The longitudinal MS lesion data resource is the enabling dataset. It is the only resource in our file with two independent raters at every time point. Its near-annual spacing is also our main constraint: irregularity has to be constructed by pairing non-adjacent visits.
Research Program
The model keeps three quantities for each patient. The first is a latent true lesion mask $y_t$ at each scan time $t$. The second is a per-rater observation model $C_r(x_t)$, a voxel-wise confusion matrix that depends on the image $x_t$ and rater $r$ but not on time. The third is a transition model that gives, for each voxel, the probability that the true label switches between scans at $t$ and $t' = t + \Delta t$. This switch probability is conditioned on features from both images and on a learned interval embedding $\phi(\Delta t)$.
For a pair of manual masks $s^r_t$ and $s^{r'}_{t'}$, the training objective is the marginal likelihood:
$$ \log p(s^r_t, s^{r'}_{t'} \mid x_t, x_{t'}) = \log \sum_{y_t, y_{t'}} C_r(s^r_t \mid y_t, x_t)\, T_{\phi(\Delta t)}(y_{t'} \mid y_t, x_t, x_{t'})\, C_{r'}(s^{r'}_{t'} \mid y_{t'}, x_{t'})\, p(y_t \mid x_t). $$
The sum runs over the latent labels at each voxel and factorizes per voxel given the network outputs, so it is cheap. The transition term $T$ is the voxel-wise label-switch distribution. We add the trace penalty of Zhang et al. on $C_r$ to discourage the model from explaining everything as rater noise.
Identification comes from three kinds of pairs. Pairs with $r \neq r'$ and $\Delta t = 0$ (both raters on one scan) constrain only $C$. Pairs at $\Delta t > 0$ also exercise $T$. Same-rater pairs across time help separate a rater's systematic bias from their visit-to-visit inconsistency. As a design choice we can test, we constrain the background-to-lesion switch rate to be non-decreasing in $\Delta t$ for small intervals. We compare three forms of $\phi$: a learned network on $\Delta t$, a sinusoidal encoding, and a single linear-in-$\Delta t$ rate.
The backbone is a shared 3D U-Net applied to patches, with features fused per pair. Applying one backbone per scan with weights shared across scans avoids the cost of backpropagating through an entire volumetric sequence, which the file flags as prohibitive. Training uses all within-patient pairs of visits rather than full sequences.
Before training anything, we compute an empirical disagreement curve on the MS data. This is the Dice between rater-$r$ mask at visit $i$ and rater-$r'$ mask at visit $j$, after registration, as a function of $|t_j - t_i|$. If its intercept is not clearly below its long-interval values, the decomposition has nothing to find, and we will report that.
Baselines
The naive reference is a static 3D U-Net trained on each scan independently against the union of both raters' masks. Lesion change is read off by subtracting its masks across visits. Every other method must beat its precision for new-lesion calls.
The strongest existing longitudinal method is the multitask early-fusion network of Denner et al.. That paper reports an Overall Score of 0.745 on its own 70-patient single-rater cohort, against 0.731 without the registration decoder and 0.710 to 0.718 for static networks. We found no reported result on Lacuna for any method on the longitudinal MS lesion data resource. We will therefore reimplement Denner's network on that data with the same backbone width, patch size, augmentation, and hyperparameter search budget as ours, trained once per rater and once on the union. Beating it means a higher Overall Score under the resource's rater-normalized protocol and higher new-lesion precision at matched recall.
The strongest disagreement baseline is Zhang et al. run per time point. This is exactly our model with the transition head removed, so it isolates the value of the temporal coupling. A second disagreement baseline is Pionono per time point. It is reported at κ 0.758 on Gleason 2019, a histopathology task that does not transfer, so we cite it only for context. A third is the per-time-point model with the fixed-penalty consistency loss of Local Spatiotemporal SSL added.
Two ablations isolate the key ingredient. In the first, $\Delta t$ is fed as a conditioning input but the factorization is removed: one head predicts each rater's follow-up mask directly. In the second, $\phi$ is replaced by the visit index. All variants get equal GPU hours and equal tuning trials, and use five-fold patient-level cross-validation, because the cohort is small.
Why not just…?
Why not just use visit indices, since this dataset is roughly annual? This is a real threat on the MS resource taken alone: with near-uniform spacing, index and interval nearly coincide. The index ablation settles it under two conditions where they come apart. One is non-adjacent visit pairs, spanning one to five years. The other is the subsampled synthetic sequences described below. If the index model matches ours on both, the interval encoding is unnecessary, and we will say so. The decomposition could still stand with a crude time variable.
Why not just collapse the two raters with STAPLE and train a longitudinal model? Consensus erases the very quantity that sets the intercept, and so removes the zero-interval anchor. Zhang et al. report 14.4% lower confusion-matrix error than STAPLE on BraTS, which suggests aggregation already loses rater structure cross-sectionally. We include the consensus-trained Denner network precisely to measure whether this matters for change detection.
Why not just feed $\Delta t$ into a per-rater network and skip the latent-truth machinery? This is the first ablation, and it may match us on per-rater Dice. Our claim is narrower: without the factorization, the model has no separate estimate of real change. Its change maps should therefore inherit rater noise, which shows up as lower new-lesion precision. If that gap does not appear, the factorization does not earn its complexity.
Why not just fit a linear rate, noise plus a constant times $\Delta t$? It may suffice. We include it as one of the three forms of $\phi$ and prefer it if the learned encoding does not improve held-out-interval likelihood.
Proposed Experiments
The first experiment is a controlled synthetic study where the right answer is known. We take OASIS3 subjects from the Local Spatiotemporal SSL setting and resample their visits at irregular intervals. We then generate several simulated raters by applying rater-specific boundary dilation or erosion biases plus random per-reading jitter to the reference labels. Because the injected rater noise and the true anatomical change are known, we measure recovery directly. One check is the error of the estimated confusion matrices against the injected noise. The other is the correlation between the predicted switch probability and the true change at each interval. This is also where visit index and interval differ most, so the index ablation is decisive here.
The second experiment uses the real MS resource with both raters. We report the resource's metric suite: Dice, positive predictive value, true positive rate, the lesion-level false and true positive rates, absolute volume difference, surface distance, and the rater-normalized Overall Score. We report these per rater at each time point, together with generalized energy distance to the two-rater pair. The new metric is new-lesion detection. A lesion counts as real change when both raters mark it at the later visit and neither marks it at the earlier one. A lesion counts as ambiguous when the raters disagree, and we track ambiguous lesions separately rather than scoring them as errors.
The decisive checks are three. First, the model's predicted disagreement at $\Delta t = 0$ must match the measured inter-rater disagreement on the same scans; an intercept fitted far from it means the decomposition is not identified. Second, trained only on pairs up to two years apart, the model must keep calibrated switch probabilities on three-to-four-year pairs, where the visit-index model has no principled answer. Third, new-lesion precision must exceed both the consensus-trained Denner network and the per-time-point Zhang model at matched recall. If the factorized model loses the third check to the plain $\Delta t$-conditioned ablation, the central claim fails.
References
- Detecting Abnormal Operations in Concentrated Solar Power Plants from Irregular Sequences of Thermal Images
- Time After Time: Deep-Q Effect Estimation for Interventions on When and What to do
- DuETT: Dual Event Time Transformer for Electronic Health Records
- TIM: A Time Interval Machine for Audio-Visual Action Recognition
- Harnessing the power of longitudinal medical imaging for eye disease prognosis using Transformer-based sequence modeling
- Disentangling Human Error from Ground Truth in Segmentation of Medical Images
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Spatio-Temporal Learning from Longitudinal Data for Multiple Sclerosis Lesion Segmentation
- Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks
- Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts
- Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling
- Longitudinal multiple sclerosis lesion segmentation data resource