Patients with brain tumors or neurodegeneration get scanned at irregular times. One might be scanned three months after surgery, then nine months later, then two weeks after that because a symptom appeared. Clinicians care less about what the next scan will look like pixel by pixel than about geometric questions: how large the lesion will be, where it will grow, and whether it is shrinking or expanding. Current methods split into two camps, and neither answers these questions well. Models that keep the full 3D scan treat time as smooth flow or as a pair of fixed timepoints. Models built for irregular timing first compress the scan into a few dozen numbers, which discards where change happens.
We propose a forecaster with three parts. Its memory is a coarse 3D grid of learned features, roughly a 24-voxel-per-side map with a few dozen channels, which keeps spatial layout but is small enough to evolve in time. Between scans, the grid drifts under a learned continuous dynamics. Whenever a new scan arrives, the grid is instantly corrected, or "jumps." Its outputs are segmentation masks and volumes at any requested future time, not reconstructed images. The claim is that keeping spatial structure in the memory while scoring forecasts in geometry yields better lesion-volume, trend-direction, and future-mask forecasts than either collapsing scans to scalars or generating full images. We also expect the gain over simple pairwise models to appear specifically when patients have more than two prior scans.
What exists and what is missing
The closest work is CRONOS. It learns a continuous velocity field over full 3D volumes by flow matching (a generator trained to transport one image toward another along a learned flow) and conditions on real-valued timestamps of several prior scans. On the Lumiere glioma cohort it reports NRMSE 7.55 against 8.38 for simply repeating the last scan. CRONOS motivates our proposal in two ways. First, it shows continuous time conditioning helps in 3D. Second, its authors acknowledge that pixel fidelity need not track clinically meaningful measurements. We differ in three respects. We forecast geometry rather than intensity. We hold memory in a coarse latent grid rather than at full resolution. And we fold in each new scan with an explicit jump update rather than stacking all context into one conditioning input.
TimeFlow conditions 3D registration on continuous time to extrapolate brain aging. Because it represents change as pure deformation, it cannot express new enhancement or a resection cavity, which a learned latent update can. 4D-DANI-Net simulates atrophy slice by slice and then needs a profile-consistency loss and 3D super-resolution to repair through-plane jitter. That is evidence against 2D decompositions and constrains our design to native 3D.
On the irregular-time side, Neural Jump ODEs supply our core mechanism: continuous latent drift between observations plus a jump network at each observation. They prove convergence to the optimal conditional expectation, but only on low-dimensional series such as PhysioNet and USHCN. CNODE applies a neural ODE to irregular Parkinson's visits but compresses each MRI to 68 cortical thickness values. CASHformer models irregular ADNI visits using only a hippocampal mesh. Both represent the "collapse first" camp we must beat. Multi-time attention constrains us: it reports Latent ODEs over 100 times slower per epoch than attention over reference times. That is why our ODE runs only on a coarse grid, never on full-resolution features.
Finally, the longitudinal MRI foundation-model benchmark shows models localize change well (about 97%) but quantify volume poorly (under 16%), with Trend-F1 of 0.19–0.63. This is the measurement gap we target.
Core Claim
Let $z(t)$ be the latent grid, with shape $C \times 24^3$. Between scans it evolves as $dz/dt = f_\theta(z, t - t_{\text{last}})$, where $f_\theta$ is a small 3D convolutional network and $t_{\text{last}}$ is the time of the most recent scan. When scan $x_i$ arrives at time $t_i$, the grid is reset:
$$z(t_i) = g_\phi\big(z(t_i^-),\, E(x_i),\, \Delta t_i\big)$$
Here $z(t_i^-)$ is the drifted state just before the scan, $E$ is a frozen pretrained 3D encoder pooled to the grid, $g_\phi$ is a small 3D convolutional network implementing the jump, and $\Delta t_i$ is the elapsed time since the previous scan.
At any query time $t$, a decoder produces a lesion or structure mask. It sees $z(t)$ and full-resolution skip features from the last observed scan, so boundaries stay sharp while the latent decides how the region has moved and resized.
Training uses masks from future scans as targets. We supervise with Dice plus volume loss at held-out real timepoints, and we generate masks on unlabeled scans using a 3D segmentation model such as SegVol or nnU-Net. The specific bet is that a spatial memory with jump updates captures where growth or regression is heading, which neither a scalar trajectory nor a single-pair model can represent.
Research Program
We will build the model on two cohorts with clinically meaningful geometry. The first is Lumiere, with sparse glioblastoma MRI over years, treatment-related abrupt changes, and irregular intervals. The second is ADNI, where hippocampal and ventricular volumes change slowly and patients average 3.32 scans over 72 months, as reported for the CASHformer cohort. For transfer to lesion tracking in CT, we will also use the autoPET/CT IV Task 2 serial whole-body scans.
The analysis has three threads. The first asks whether a spatial latent beats scalar collapse at equal information. The second asks whether jump-based filtering helps as the number of prior scans grows. The third asks whether scan timing itself carries information. NJ-ODE's consistency proof assumes timing is independent of disease state, yet clinical scans are often ordered because something worsened. We test this by giving the jump network the interval and comparing against a version trained with intervals randomly permuted within patients.
Baselines
The naive reference is persistence: the last observed mask carried forward unchanged, plus a variant that linearly extrapolates the last two volumes. CRONOS confirms that repeating the last image beats ConvLSTM and SimVP on sparse series. Any method that fails to beat persistence on volume error and future-mask Dice has learned nothing. Beating it means lower median absolute volume error with paired confidence intervals excluding zero, and higher Dice on masks at future timepoints.
The strongest full-volume method is CRONOS Continuous. Its forecast images will be segmented by the same frozen segmenter used for our targets, so both systems are scored in the same mask space. We will retrain it on our exact splits, matching GPU-hours and running the same hyperparameter search budget. Its reported Lumiere NRMSE is 7.55 versus 8.38 for last-image repetition. No reported result was found on Lumiere for any mask-level or volume-forecast metric, so we will establish that number ourselves. Beating CRONOS means better volume and trend accuracy at equal or lower compute; matching its NRMSE is not the goal.
The strongest collapse-first competitor segments every scan, extracts lesion or structure volumes plus regional summaries, and runs NJ-ODE and mTAN on those scalars. The regional summaries are cortical thicknesses in the CNODE style and per-region lesion volumes. This baseline has the same masks available and the same tuning budget.
The strongest pairwise competitor is a 3D residual U-Net in the style of the LongiSeg extension. It receives the last scan, the last mask, and the elapsed time as an added input channel, and predicts the future mask directly. On autoPET Task 2, the LongiSeg extension reports Dice 63.71 with synthetic longitudinal pretraining, versus 53.06 cross-sectionally. Those figures are for segmenting an observed follow-up, not forecasting, so they bound difficulty rather than serving as a target.
Ablations isolate the key ingredient in three ways. We replace the 3D grid with a pooled vector of equal parameter count. We remove the continuous drift so the model jumps only. And we permute intervals to test whether timing carries information.
Why not just…?
Why not just segment each scan and model the volumes with NJ-ODE? This is the real threat. For a single total-volume number it may well tie us, because tumor volume trajectories are low-dimensional. The proposal settles it directly. If the scalar model matches us on total volume but we win clearly on future-mask Dice and on predicting which region grows, then the spatial latent earns its place for localization but not quantity, and we will report it that way. If the scalar model wins on both, the central claim fails.
Why not just run CRONOS and segment its output? Flow matching optimizes image fidelity, and its authors note fidelity need not track clinical measures. Its velocity-field transport also has no explicit mechanism for abrupt events such as resection. The head-to-head comparison in mask space answers this. We will also stratify Lumiere intervals by whether surgery or treatment changes occurred between scans, where we expect the jump update to matter most.
Why not just give a pairwise 3D U-Net the time gap as an input? With only one prior scan, our model reduces to almost exactly this. Its only additional value comes from integrating several irregular scans. We therefore report results stratified by number of prior scans. If the gap over the pairwise model does not widen beyond two scans, the filtering machinery is unjustified.
Why not just slice into 2D and use a large pretrained model? The longitudinal MRI benchmark shows multi-view 2D inputs yield under 16% volume-quantification accuracy and can degrade temporal reasoning in smaller models. 4D-DANI-Net needed extra machinery to repair slice inconsistencies. We therefore do not include a 2D variant as a primary competitor, beyond a single slice-wise ablation of the encoder.
Proposed Experiments
The decisive checks are four.
On Lumiere, we forecast the next-scan tumor mask from all prior scans. We report Dice, absolute volume error, and Trend-F1 for growth versus regression, using patient-level cross-validation because the cohort is small.
On ADNI, we forecast hippocampal and ventricular volume and masks at the actual next-visit time and at visits two steps ahead. We report the same metrics, and test zero-shot on OASIS-3 to check scanner robustness.
Across both cohorts, we plot each method's error against the number of prior scans and against interval length. We also report the permuted-interval ablation. If ordering scans is informative, scrambling intervals should measurably raise volume error on Lumiere but much less on ADNI, where visits follow protocol. That contrast would be the first direct test of informative sampling on 3D data.
Finally, we report wall-clock time and peak memory against CRONOS to show the coarse-grid ODE stays tractable. We will feed forecast volumes into the benchmark's volumetric-quantification questions for the Lumiere and ADNI subsets to see whether explicit geometric forecasting closes part of the gap that vision-language models leave open.
References
- CRONOS: Continuous time reconstruction for 4D medical longitudinal series
- Temporal Conditioning for Longitudinal Brain MRI Registration and Aging Analysis
- Degenerative Adversarial NeuroImage Nets for 3D Simulations: Application in Longitudinal MRI
- Neural Jump Ordinary Differential Equations: Consistent Continuous-Time Prediction and Filtering
- Conditional Neural ODE for Longitudinal Parkinson's Disease Progression Forecasting
- CASHformer: Cognition Aware SHape Transformer for Longitudinal Analysis
- Multi-Time Attention Networks for Irregularly Sampled Time Series
- How Good are Foundation Models in Longitudinal MRI Disease Progression Reasoning?
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation
- Promptable Longitudinal Lesion Segmentation in Whole-Body CT