The Report Is the Free Label: Re-fitting Chest X-ray Models to the Present Using Their Own Radiology Reports

Medical imaging models lose accuracy after deployment as scanners, protocols and diagnostic habits change. Fixing this usually requires freshly labeled exams from the current period, and hospitals can rarely afford more than a few dozen. Every new chest X-ray, however, already comes with a free-text radiology report written at the same moment by the radiologist who read it. This proposal treats that report as supervision that is free, current and plentiful. The model is periodically re-fit to the most recent months of image–report pairs, and the handful of expert labels is kept for choosing and calibrating the model rather than for training it. The report is not fed in as raw prose. It is first parsed into a small graph of findings and their locations, using RadGraph, a schema that turns a report into anatomical sites, clinical observations and relations between them (RadGraph). The graph keeps what was seen and discards how it was phrased. Reporting style drifts over the years too, and we do not want the image model to chase it.

In two sentences: under temporal drift, the scarce resource is expert labels, not recent data, because each recent exam arrives paired with a report. We propose re-aligning a vision-language model's image side to recent reports expressed as structured findings graphs, and test whether this recovers the ranking ability that recalibration and domain-invariance methods are known not to recover.

Core Claim

The claim has three parts. First, image-only chest X-ray classifiers backtested chronologically on MIMIC-CXR lose discrimination (AUROC, the ability to rank sick above healthy patients), not just calibration. Second, re-fitting the image encoder to recent-era image–report pairs recovers a meaningful share of that loss using at most 100 labeled exams per era. Third, encoding the report as a RadGraph findings graph beats two alternatives: free-text alignment, which absorbs stylistic drift, and fixed-vocabulary labels from the rule-based CheXpert labeler, which cover only 14 findings and miss rare ones. We expect the advantage of the graph over labeler labels to show up mainly on rare, long-tail findings. If it shows up nowhere, the structure is unnecessary and we will say so.

At prediction time the model sees only the image. The report for the current exam does not yet exist when a prediction would be useful. Keeping text out of inference also sidesteps the tendency of vision-language models to over-trust accompanying text documented in Words or Vision. Reports enter only as training signal from exams already read.

Where This Sits

Three results in the literature motivate the design, and two constrain it.

Three results motivate it. Evaluating Model Performance in Medical Datasets Over Time backtests models chronologically on six medical datasets, including MIMIC-CXR. It shows that random splits overstate performance, that high-capacity models go stale faster, and that recent sliding windows beat all-history training when drift is fast. That work treats images unimodally and never uses the reports. Systematic Review of Approaches to Preserve ML Performance under Temporal Dataset Shift finds that recalibration fixes miscalibration but does not restore discrimination. The only feature-level fix that preserved discrimination was expert grouping of drifting codes. A RadGraph findings graph is effectively an automatic version of that grouping for radiology text, and nobody has tested it under drift. Model Assessment and Selection under Temporal Distribution Shift gives an adaptive rolling window, a tournament that picks how far back to look by comparing losses on fresh data. It was tested on housing prices and topic trends. We use it to choose the report window, and the free reports supply the fresh evaluation data the tournament needs.

Two results constrain it. Evaluation of Domain Generalization and Adaptation under Temporal Shift shows on a decade of MIMIC-IV data that no domain-generalization or unsupervised adaptation method beat plain pooled training. Forcing features to be invariant across time threw away useful signal that changes over time. Our method therefore enforces no invariance; it re-fits to the present. Limitations of Synthetic Data Generation in Specialized Data-Scarce Domains finds that generated images are idealized and rarely beat simple AugMix augmentation, so we use no generative augmentation.

Two works supply the method. Style-Aware Radiology Report Generation shows that serialized RadGraphs separate clinical facts from radiologist style, raising RadGraph F1 from 0.186 to 0.228 in generation. We use the same separation in the reverse direction, as a training target for the image encoder, and add a time axis the paper never examines. LLaVA-Rad supplies the image encoder, BiomedCLIP-CXR, trained on 697K image–report pairs and evaluated only cross-sectionally. GraphCare shows that structured knowledge graphs let 36 patients match 720 on EHR tasks. It also warns that noisy extracted relations hurt when samples are few, which is why we use only report-internal RadGraph relations rather than external multi-hop knowledge.

Research Program

We first reproduce the drift. Following EMDOT's chronological protocol on MIMIC-CXR, we divide exams into consecutive eras. A classifier is trained on the earliest era and tested on each later one. For each later era we measure AUROC, average precision and calibration error, all on the image alone. This establishes how much discrimination, as distinct from calibration, is lost.

We then characterize where the drift lives. We track three quantities across eras. Image-embedding drift is the distance between era means in the frozen encoder space. Surface-text drift is the change in report n-gram distribution. Findings drift is the change in RadGraph entity and relation frequencies. If surface text drifts much more than findings, free-text alignment should import noise that graph alignment avoids. That is a checkable prediction made before any adaptation result is seen.

The adaptation method keeps the language side frozen. RadGraph graphs are extracted with the published extractor, which reports 0.94 entity F1 and 0.82 relation F1 on MIMIC-CXR. Each graph is serialized to a string and embedded with a frozen text encoder. We then update only a small adapter on the BiomedCLIP-CXR image encoder, using a contrastive loss that pulls each recent image toward its own graph embedding and away from others in the batch. The adaptive rolling window chooses how many past months of pairs to include. Its tournament is scored on held-out recent pairs, which cost no labels. Finally, the 25 to 100 labeled current-era exams fit a light classification head and a temperature for calibration, and nothing else.

Baselines

The naive reference is the model trained on the earliest era and deployed unchanged. Every other number is read as recovered AUROC relative to it.

Pooled training on all historical labeled data is the strongest existing method on this question. It is plain empirical risk minimization, run with the same encoder, augmentation (AugMix) and tuning budget as our method. Guo et al. found it beat every robustness algorithm across MIMIC-IV, though sepsis AUROC still fell by up to 0.090 (Evaluation of DG/UDA under temporal shift). No reported result found on Lacuna for chronologically split chest X-ray AUROC; EMDOT reports its MIMIC-CXR trends without per-split numbers in the file. We will therefore regenerate the EMDOT all-history and sliding-window baselines ourselves. For scale on non-chronological splits, CXR-LT 2024 reports a top mAP of 0.281 on its noisy long-tailed test set, 0.526 on the gold-standard 26-class set, and 0.129 for zero-shot unseen classes (CXR-LT 2024). We expect chronological numbers to be lower. Beating pooled training means a higher per-era macro AUROC, with a paired bootstrap confidence interval excluding zero, in at least the two most recent eras, at a label budget of 100 or fewer.

The EMDOT sliding-window model is trained only on recent labeled data. It tests whether recency alone, without reports, is enough.

The labeler pseudo-label baseline is the most direct competitor to our design choice. It runs the CheXpert labeler on recent reports and fine-tunes the same adapter on the resulting 14 noisy labels, with the same window selection and compute. It uses the current reports without any graph structure.

The key ablation holds everything fixed and swaps only the report encoding: serialized RadGraph, raw report text, or labeler labels. A second ablation replaces recent pairs with an equal number of old-era pairs. This separates the benefit of recency from the benefit of more data. The last reference is post-hoc temperature scaling of the naive model on the labeled exams.

Why not just…?

Why not just recalibrate the old model on the few new labels? The systematic review finds recalibration restores calibration but consistently fails to restore discrimination (Guo et al. review). Rescaling scores cannot re-order patients whose image features have shifted. We include it anyway. If it matches our method on AUROC, the drift on chest X-rays is mostly a calibration problem, and that is a useful negative result.

Why not just pool all history, as Guo et al. recommend? Pooled training is the right baseline and may be hard to beat on common findings. EMDOT shows it loses to recent windows when drift is fast (EMDOT). Our comparison is designed to show which regime chest X-ray drift falls in. It does so per era and per finding, not as one averaged number.

Why not just fine-tune on labeler pseudo-labels from recent reports? This is a real threat. It uses the same free signal with no graph machinery, and the labeler is accurate on common findings (F1 0.973 for pleural effusion, per MIMIC-CXR). Our bet is that it fails on rare findings outside its 14 classes, and on phrasings whose meaning drifts while its fixed rules do not. The key ablation settles this. If labeler labels come within confidence intervals of the graph on long-tail classes too, we drop the graph and report that the simpler method suffices.

Why not just fit a linear probe on the frozen foundation encoder with the few labels? This is cheap and resists overfitting, which matters given EMDOT's finding that complex models go stale faster. But a probe cannot move features that the hardware or protocol change has corrupted. We run it at every label budget. If it matches adapter re-alignment, image-side re-fitting is unnecessary.

Proposed Experiments and Decisive Checks

All methods share the chronological MIMIC-CXR splits, the BiomedCLIP-CXR encoder, identical compute, and a matched hyperparameter search. Label budgets are 0, 25, 50 and 100 labeled exams per era. Metrics are per-era macro AUROC on the 14 standard findings, average precision on the CXR-LT rare classes present in each era, and expected calibration error.

There is one circularity risk. Evaluation labels in MIMIC-CXR are themselves derived from reports, so report-trained methods could look good simply by fitting labeler quirks. We address this in two ways. We score on the CXR-LT gold-standard subset wherever its exams fall within an era. We also check that the method ranking is unchanged when evaluation uses labels from a different extractor than the one used in training.

Three results decide the proposal. First, graph-based re-alignment must recover AUROC that temperature scaling does not. Second, it must beat labeler pseudo-labels on rare findings. Third, both gains must hold on report-independent labels. If the first fails, the drift is a calibration problem and the paper becomes a drift-characterization study. If only the second fails, the recommendation becomes "use the free reports, skip the graph."