Catching slow scanner degradation before it breaks a clinical AI model

Clinical imaging models are usually checked for domain shift the way a hospital checks a new scanner: compare a batch of scans from "here" with a batch from "there" and ask whether they differ. Real scanners do not fail that way. A detector loses gain unevenly, or a receive coil weakens by a fraction of a percent each week. The images a deployed model sees drift slowly, inside one machine, while the patient population is also changing. Industrial fault detection has spent decades on exactly this problem, which it calls incipient faults: degradation that stays below the noise floor until it is nearly catastrophic. That field's main tool is the residual. You predict what a healthy system should output given its known operating conditions, then accumulate the gap between prediction and observation over time, rather than thresholding raw outputs.

We propose to carry that tool into medical AI monitoring. The monitor predicts what a deployed model's feature and score statistics should look like given who is being scanned (age, sex, inpatient status, protocol). It also compares each scanner against peer scanners at the same site that see a similar population. It then accumulates the unexplained part over time with a sequential change test designed for slow ramps. The claim is that this residual monitor detects gradual, scanner-specific acquisition drift with far fewer scans than batch two-sample tests. It should also stay quiet when the change is demographic or epidemiological, a distinction current medical drift detectors cannot make at small sample sizes.

Core Claim

Two medical drift papers motivate this proposal by exposing the gap. Empirical data drift detection experiments on real-world medical imaging data monitored 239,235 longitudinal chest radiographs. It used maximum mean discrepancy (MMD, a kernel two-sample test) on autoencoder features and model outputs. At 500 images per window, p-values rarely fell below 0.1, and reliably detecting a 40% shift in the proportion of male patients needed about 4,000 images. Automatic Dataset Shift Identification to Support Safe Deployment of Medical Imaging AI separates prevalence shift from covariate shift using a combined output-plus-feature detector. It works on static batches of 500 to 1,000 scans, so it cannot say whether a covariate shift comes from the machine or from the patients.

Neither paper conditions on the patient context it already records, and neither accumulates evidence across time. The incipient-fault literature supplies both mechanisms. Residual-aware health prediction of power transformers via spatiotemporal graph neural networks builds a health indicator from prediction residuals, with quantile thresholds and temporal continuity filtering, and shows that this indicator rises before failure. Its evaluation is only on synthetic SCADA data. Unsupervised anomaly detection in spatio-temporal stream network sensor data shows that separating one sensor's drift from shared environmental variation requires modelling the network of neighbouring sensors. We reuse both ideas directly. Patient covariates play the role of operating conditions, and peer scanners play the role of neighbouring sensors.

What is new is not sequential testing on its own. It is the residualization against patient context and peer scanners, which is what should make low-N detection possible without confusing population change for hardware change. If plain sequential testing on unconditioned statistics does just as well, the claim fails, and the experiments are built to show that.

Research Program

The monitor works on any frozen deployed model. For each scan $t$ it extracts a low-dimensional statistic $z_t$. This is a projection of self-supervised features plus the model's output score, the same signals used by the batch detectors above, so the comparison is like-for-like. A small regression $g$, fit on a reference period, predicts $z_t$ from patient and protocol covariates $c_t$. When peer scanners exist, the shared component estimated from them over a rolling window is also removed:

$$r_t = z_t - g(c_t) - \bar{r}^{\,\text{peer}}_t ,$$

Here $r_t$ is the residual for scan $t$, and $\bar{r}^{\,\text{peer}}_t$ is the mean covariate-adjusted residual of the other scanners at the same site over the recent window. Evidence accumulates with a one-sided cumulative-sum (CUSUM) statistic on the whitened residual:

$$S_t = \max\!\big(0,\; S_{t-1} + \hat{w}_t^\top \Sigma^{-1/2} r_t - k\big),$$

$\Sigma$ is the reference residual covariance. $\hat{w}_t$ is a running estimate of the drift direction, which matters because a slowly growing fault keeps pointing the same way. $k$ is a slack constant, and an alarm fires when $S_t$ exceeds a threshold $h$. Both $k$ and $h$ are set by simulation on the reference period to hit a target false-alarm rate. The reason to expect a gain is simple. Removing variance explained by covariates and peers shrinks $\Sigma$, and CUSUM delay for a given drift size grows with the noise-to-signal ratio. Any variance removed therefore translates directly into earlier alarms.

The program has three parts. The first builds physically grounded incipient faults, since no public dataset labels slow scanner wear. For MRI we inject them in raw k-space from fastMRI and FastMRI Prostate: progressive loss of signal-to-noise in one receive coil, a slowly growing phase ramp of the kind field drift produces, and gradual gradient timing error. Faulty data are then reconstructed with the provided reconstruction pipeline. This works below the reconstruction step, where Reproducibility and quality assurance in MRI locates real "image quality drift." For radiographs and mammograms we inject image-domain ramps in detector gain non-uniformity, noise, and sharpness loss. Each fault grows linearly or exponentially over a simulated deployment of several thousand scans.

The second part builds realistic nuisance changes, so that false alarms are tested against something harder than stationary data. These are covariate resampling by age, sex, and inpatient status, following the protocols of the two drift papers, and prevalence surges. The third part fits and evaluates the monitor. The main outcomes are detection delay at a fixed false-alarm rate, attribution between hardware and population, and how much downstream accuracy has already been lost by the time the alarm fires.

The active-probing idea from Bayesian Diagnosability and Active Fault Identification is left out on purpose. That paper reports passive misidentification above 45% and near-zero misidentification with active probing. Designing active probes, meaning scanner test sequences, needs scanner access we cannot assume, and the paper itself treats only abrupt faults. Passive residual monitoring is the step that has to work first.

Baselines

The naive reference point is monitoring the deployed model's running AUROC and its mean output score against static thresholds. This is what many deployments do. The empirical drift study already found that AUROC stayed flat through COVID-19 onset and demographic shifts, so we expect it to be late or silent. It is included to measure how much accumulated damage occurs before it reacts.

The strongest existing detector is the Duo detector from Automatic Dataset Shift Identification. It combines output-distribution tests with MMD on self-supervised features under Bonferroni correction, and it reports covariate acquisition shift detection above 80% at 500 test images. We run it on sliding windows of 100, 250, 500, and 1,000 scans over the same simulated streams. It uses the same encoder and the same reference set of about 2,000 images, and its window size and significance level are tuned on the same reference period to the same false-alarm rate. Its shift-identification stage serves as the attribution baseline. We also run the TAE+BBSD combination from the empirical drift study, meaning an autoencoder-feature test combined with a black-box shift test on model outputs.

The strongest simple competitor is a plain CUSUM on the same $z_t$ without covariate or peer residualization, tuned by the same simulation procedure. Two ablations isolate the key ingredient: covariate conditioning only, and peer contrast only. Beating the baselines means two things. First, a median detection delay at most half that of the best-tuned Duo window and of plain CUSUM, at matched false-alarm rate, on at least the MRI and mammography fault families. Second, attribution accuracy under simultaneous population shift clearly above Duo's identification stage.

For downstream correction, Unsupervised Prediction Alignment (UPA) maps shifted score distributions back to the reference distribution. It reports raising Youden's index on mammography Scanner B from 0.295 to 0.651 with 250 to 500 unlabeled scans. It is the reference for what reactive correction recovers after an alarm.

Why not just…?

Why not just run CUSUM or a short-window MMD on the model's scores? This is the real threat. Sequential testing alone may capture most of the gain over batch tests. The plain-CUSUM baseline and the two ablations settle the question. If unconditioned CUSUM matches the full monitor on delay and attribution, the residualization adds nothing, and we will report that the contribution reduces to applying sequential testing to medical drift. We expect conditioning to matter most in exactly the cases the empirical drift study found hardest: shifts in older and inpatient cohorts needed about 30% changes to register, because they move feature statistics a lot without any hardware cause.

Why not just recalibrate scores with UPA whenever they move? UPA assumes disease prevalence is constant. Its authors also state that it cannot restore performance when the shift destroys discriminability. That is the regime of the prostate MRI drop from 0.791 to 0.505 AUC in The Impact of Scanner Domain Shift. Recalibrating during a prevalence surge would also mask a real clinical signal. We test this directly by running UPA continuously on the fault-plus-prevalence streams and measuring its calibration error against an alarm-then-intervene policy.

Why not just rely on monthly phantom QA? Phantom QA measures the scanner, not the model's sensitivity to it, and it runs only between long gaps. In our simulations we place a QA check at fixed intervals with an idealized perfect detector, and count the scans processed while the model's AUC is already degraded. If ramps are slow relative to the QA interval, QA wins and our monitor is unnecessary. The fault-rate sweep reports where that crossover lies.

Why not just make the model robust with noise augmentation? The scanner-shift study found that Gaussian noise augmentation did not close scanner gaps and often hurt in-domain accuracy. Robustness also does not tell anyone the scanner needs service. We do not rerun this comparison.

Proposed Experiments

The decisive MRI experiment trains a slice-level PI-RADS classifier on FastMRI Prostate. It then streams held-out patients, resampled into a long deployment sequence, with k-space coil-SNR, phase-drift, and gradient-timing ramps beginning at a random time. We report detection delay, the classifier's AUC at alarm time, and false alarms per 10,000 scans. Since FastMRI Prostate comes from a single vendor, "peer scanners" are simulated as parallel streams of the same population, only one of which degrades. This is an optimistic setting for the peer term, and we say so.

The cross-modality experiments repeat the setup on OPTIMAM mammography across its four vendors and on RSNA/PadChest radiographs, with image-domain ramps. The false-alarm experiment is the hardest one. Streams contain only demographic and prevalence shifts, using the resampling protocols of both drift papers and, where access allows, the Trillium COVID-19 onset. The monitor should attribute these to population, not to hardware.

Finally, a mixed experiment overlays a hardware ramp on a simultaneous population shift, which is where every baseline is expected to confuse the two. Published comparisons stop at 500 scans or more because they report no lower windows; no reported result found on Lacuna for any detector below 500 images per window. Our delay curves will fill that range. A negative result is still informative: if residualization helps only when peer scanners exist, the method suits multi-scanner sites and not single-scanner clinics.