Radiologists read a chest X-ray by comparing it with the patient's earlier films and asking what changed. A new family of "longitudinal" models copies this habit: they take a prior and a current image and predict whether a finding is improving, stable, or worsening. They all quietly assume that any visual difference between the two images is biological. That assumption fails in real hospitals. Scanners get upgraded, image processing software changes, and a patient imaged standing in radiology one year may be imaged lying in an ICU bed the next. In each case the pixels change even when the lungs have not.
This note makes two linked claims and proposes a test of each. First, nobody has measured how comparison models age when they are deployed forward in calendar time, rather than scored on random splits. Second, hospitals already record a large, free source of supervision for separating equipment change from disease change: pairs of studies whose reports say "unchanged" but whose acquisition differs, by view, device, or era. We propose to train the part of the model that encodes differences between the two images to be insensitive to acquisition only on those pairs. This avoids making the whole representation time-invariant, which is the design that prior work found backfires.
Core Claim
The difference signal in a longitudinal model should be invariant to acquisition changes between the two images. It should not be invariant to calendar time in general.
The proposed mechanism is a targeted consistency objective. Consider a pair of images that the radiology report labels as stable but that was acquired differently. This could be a posterior-anterior (PA) film versus a bedside anteroposterior (AP) film, a different device manufacturer in the DICOM header, or acquisitions years apart. For such pairs, the model's predicted change and its internal difference features are pulled toward those of stable pairs acquired the same way. Pairs labeled improving or worsening are trained only with the ordinary progression loss.
Writing $d(x_p, x_c)$ for the difference embedding of prior image $x_p$ and current image $x_c$, the objective is
$$ \mathcal{L} = \mathcal{L}_{\text{prog}} + \lambda \, \mathbb{E}_{(x_p,x_c) \in S_{\text{mis}}} \big[ -\log p_{\text{stable}}(x_p,x_c) + \| d(x_p,x_c) - \bar d_{\text{match}} \|^2 \big]. $$
Here $\mathcal{L}_{\text{prog}}$ is the standard three-way progression cross-entropy. $S_{\text{mis}}$ is the set of stable pairs with mismatched acquisition. $p_{\text{stable}}$ is the predicted probability of "stable". $\bar d_{\text{match}}$ is a running mean of difference embeddings over stable pairs with matched acquisition. $\lambda$ is a weight chosen on a temporally held-out validation era.
This objective differs from invariant-risk-style training in one important way. The single-image features are left free to use time-varying signal. Only the comparison channel is constrained, and only on examples where the report tells us nothing biological changed.
Why This Gap Is Real
Three findings motivate the proposal.
A deep learning model for the diagnosis of sacroiliitis found that a model trained on circa-2008 MRI lost accuracy on the same cohort's 5- and 10-year follow-ups. The authors attributed this to newer scanner hardware, and they had to exclude 48% of exams for protocol inconsistency. They offered no way to separate hardware change from disease change.
The authors of BioViL-T note that their comparison model breaks under portable-AP versus standing-PA discrepancies. BioViL-T splits the image representation into static and temporal parts to model progression, but it never supervises the temporal part against acquisition change.
CoCa-CXR restricts cross-attention between prior and current images to local spatial neighborhoods. It reports the best progression accuracy on MS-CXR-T, but only under time-agnostic evaluation.
On the drift side, two papers constrain the design:
- Evaluation of domain generalization and adaptation on temporal dataset shift showed on a decade of MIMIC-IV ICU data that invariance methods and domain adaptation never beat plain training on pooled data. Their explanation is that drifting features carry real signal. We take this as a negative result that rules out forcing whole-model time invariance.
- Evaluating Model Performance in Medical Datasets Over Time (EMDOT) supplies the method for evaluation. It backtests models by training on data up to a date and testing on later data, and it shows that random splits overstate performance. However, it only backtests single-image or tabular models.
Two further papers are single-image tools we treat as components or baselines rather than competitors. Unsupervised Prediction Alignment (UPA) repairs single-image score shifts from scanner changes by matching output distributions, assuming disease prevalence stays fixed. CheXstray detects drift from DICOM metadata and latent image statistics. Neither handles two images from one patient taken on different equipment.
Research Program
Stage 1: measurement. We port the EMDOT rolling-deployment protocol to paired chest X-rays. Studies from MIMIC-CXR (via MIMIC-CXR) are binned into calendar eras; because MIMIC dates are de-identified, we use the coarse real-year groups EMDOT relied on. We then train BioViL-T-style and CoCa-CXR-style comparison models on eras up to $t$ and test on later eras.
Progression training labels come from comparison sentences in reports. For this we use the CXR-4 construction built on Chest ImaGenome, as in CoCa-CXR. We report decay separately for three kinds of pairs:
- pairs whose two images share acquisition,
- pairs mismatched in view position,
- pairs mismatched in manufacturer or era.
This answers the open question in the file directly: does conditioning on a patient's own prior slow model decay, by acting as an anchor, or speed it up, by importing drift from two images instead of one? PadChest, which carries real acquisition dates for 2007–2017, is a secondary stream if enough repeat-patient studies exist. We will verify this count before committing to it.
Stage 2: the method. We add the stable-mismatch consistency objective to the strongest Stage 1 architecture. Acquisition mismatch is defined from DICOM view position and manufacturer fields. We also use the era gap between the two studies as a mismatch signal, because it catches unrecorded changes in processing software.
Stage 3: stress testing. We test under simultaneous prevalence shift, the setting where UPA-style alignment is known to be fragile. To do this, we reweight test eras toward worsening pairs in the way a respiratory surge would.
Baselines
The naive reference is a single-image classifier trained on the same era-restricted data. Progression is read off as the difference of its probabilities on the current and prior image, thresholded into three classes. It shows what comparison architectures add at all. Medical Large Vision Language Models with Multi-Image Visual Ability suggests it should lose on comparative tasks, but that has not been shown under drift.
A stronger variant of this naive baseline applies UPA-style score alignment per acquisition group before differencing. This is the most direct transplant of the existing drift fix.
The strongest existing method is CoCa-CXR's regional cross-attention model. The best reported number is 65.0% average progression accuracy on MS-CXR-T across five conditions, 4.8 points above BioViL-T, whose derived score is about 60.2% (CoCa-CXR). Both figures come from time-agnostic evaluation. We found no reported result on Lacuna for any longitudinal model under rolling temporal deployment, so Stage 1 produces the first such numbers.
For fairness, every model uses the same:
- era splits,
- report-derived labels,
- backbone initialization,
- GPU-hour budget,
- hyperparameter search size, with the search run on the held-out last training era rather than on test eras.
Patients who appear in MS-CXR-T are removed from all training sets.
We also run two training-level competitors:
- all-historical pooled training and sliding-window retraining, which the ICU study and EMDOT identify as the bar to clear;
- invariant risk minimization across eras, for completeness.
The ablation that isolates the key ingredient applies the same consistency loss to randomly chosen stable pairs, ignoring acquisition mismatch. If this matches the full method, the gain comes from extra stable-pair regularization, not from teaching acquisition invariance.
Beating the field means two things. The method must raise macro-F1 on future-era mismatched pairs over the best of CoCa-CXR and metadata conditioning. That gain must exceed seed-to-seed variation. And it must not lose more than one point on matched pairs or on the original MS-CXR-T split.
Why not just…?
Why not just subtract a static model's scores, after recalibrating each scanner? UPA-style alignment does fix single-image score shifts; it raised Youden's index from 0.295 to 0.651 on a new mammography scanner (UPA). But it requires fixed prevalence. The recalibration review (Systematic Review of Approaches to Preserve ML Performance under Temporal Shift) also found that recalibration restores calibration but not ranking ability. Subtle interval changes are a ranking problem. The Stage 3 prevalence-shift test is where we expect this baseline to fail, and it is run as a named baseline, not dismissed.
Why not just feed acquisition metadata (view, manufacturer, year) to the comparison model as input tokens? This is the real threat. It may capture most of the view-position effect. Our argument against it is narrower: metadata does not record changes in processing software or gradual hardware drift within one vendor label. The experiment settles it by splitting results into pairs whose mismatch appears in metadata and same-metadata pairs from different eras. If metadata conditioning matches our method on both, we will report that the simpler fix suffices. We also test the combination of the two.
Why not just retrain on recent data on a sliding window? EMDOT and the ICU study show that retraining is the strongest defense against temporal decay. We therefore run the method under the same retraining schedule rather than instead of it. The question is whether acquisition-invariant difference features reduce decay between retrains, measured as the slope of accuracy against months since the last retrain.
Why not just enforce time invariance with invariant risk minimization? The ICU benchmark found that it never beat pooled training, because it discards useful drifting signal (Guo et al.). We include it only to confirm that result on images. Our loss is designed around that finding: it constrains only the comparison channel, and only on report-certified stable pairs.
Proposed Experiments
The headline metric is macro-F1 across improving, stable, and worsening, evaluated on future eras and split into matched and mismatched pairs. Secondary metrics are:
- the false-change rate on stable pairs, meaning the fraction wrongly called improving or worsening;
- calibration error after era shift;
- original MS-CXR-T accuracy, for comparability with the 65.0% figure.
The first decisive check is the false-change rate on stable PA-versus-AP and cross-era pairs. If our method does not reduce it relative to CoCa-CXR and metadata conditioning, the core claim fails.
The second is a controlled perturbation test. We apply realistic processing changes to the prior image only: contrast curves, noise, and resolution typical of vendor differences. We then measure how much the predicted progression distribution moves on stable pairs. An acquisition-invariant difference channel should barely move, while baselines should move.
The third reads the Stage 1 decay slopes. If comparison models decay no faster than single-image models even on mismatched pairs, the problem is smaller than the sacroiliitis evidence suggests, and we will say so.
References
- A deep learning model for the diagnosis of sacroiliitis according to Assessment of SpondyloArthritis International Society classification criteria with magnetic resonance imaging
- Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
- CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding
- Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine
- Evaluating Model Performance in Medical Datasets Over Time
- Automatic correction of performance drift under acquisition shift in medical image classification
- CheXstray: Real-time Multi-Modal Data Concordance for Drift Detection in Medical Imaging AI
- MIMIC-CXR: A large publicly available database of labeled chest radiographs
- Medical Large Vision Language Models with Multi-image Visual Ability
- Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine