Monitoring Medical AI Without Phantom Drift from Signal Timing

Regulators increasingly want to watch deployed AI systems continuously instead of certifying them once and walking away. A statistical alarm that fires whenever a deployed model's behavior changes is the natural tool. For devices that read physiological waveforms such as electrocardiograms (ECGs), these alarms have a blind spot. Many neural networks and wavelet front-ends change their outputs when a signal is merely shifted in time by a few milliseconds. A firmware update that changes where the recording window starts, or a resampling step in a new hospital's pipeline, can therefore make a perfectly healthy device look as if it has decayed. We call this phantom drift. A monitor that cannot tell phantom drift from real decay either cries wolf or must be tuned so loosely that it misses real harm. In both cases the regulator ends up redrafting audit rules every time an acquisition pipeline changes, which is regulatory lag at the level of the audit protocol itself.

The proposal is to build continuous audits whose test statistic is invariant to timing changes by construction, whether or not the audited device is. The regulator already has query access to the device, as in consumer-side auditing. For each incoming record, it queries the device on a pre-declared set of time-shifted copies and monitors the loss averaged over that set. A change in deployment timing then cannot move the monitored statistic, while genuine changes in patients, labels, or the device still can. The nuisance set is written into the audit protocol once, so the rule survives pipeline changes without being rewritten.

Core Claim

Let $g_\tau$ shift a record $x$ by $\tau$ samples (circularly, within the recording window). Let $T$ be a declared set of nuisance shifts that forms a group, for example all circular shifts up to a fixed length. Let $f$ be the audited device and $\ell$ a loss or behavior score against the label or reference $y$. The audit monitors

$$ s(x, y) = \frac{1}{|T|} \sum_{\tau \in T} \ell\big(f(g_\tau x), y\big), $$

and because $T$ is a group, $s(g_\sigma x, y) = s(x, y)$ for every $\sigma \in T$. Any change in the distribution of deployment offsets within $T$ therefore leaves the distribution of $s$ exactly unchanged. Phantom drift is removed from the hypothesis being tested, not filtered out afterwards.

We feed $s$ into a "testing by betting" procedure. This is a sequential test that bets against the null of no change and rejects when its accumulated winnings exceed a threshold; it keeps its false alarm rate bounded no matter how long it runs. Concretely, the wealth is $W_t=\prod_{i\le t}\big(1+\lambda_i (s_i - \mu_0)\big)$, where $\mu_0$ is the reference mean of $s$ and $\lambda_i$ is a predictable bet size. The test rejects when $W_t \ge 1/\alpha$, and the false alarm bound is Ville's inequality.

The new part is not the betting test or the shift-invariant transform, both of which exist. It is the formal separation of nuisance timing change from real decay inside a regulator's black-box audit. This also yields a by-product certification quantity: the variance of $\ell$ across the orbit, a per-device shift-sensitivity index that a regulator can report alongside accuracy.

Relation to Prior Work

Two papers motivate the problem and supply the monitoring machinery. An Auditing Test to Detect Behavioral Shift in Language Models builds anytime-valid betting audits in which a regulator poses as a consumer, and reports detection up to 80% with false positive rates below 0.4. Its authors note the test cannot separate benign drift from illicit change. Our test addresses exactly that weakness, for one well-defined class of benign drift, and in waveforms rather than text. Towards a Post-Market Monitoring Framework for Machine Learning-based Medical Devices monitors medical devices with cumulative-sum (CUSUM) control charts and causal corrections for clinician feedback, and finds calibration monitoring detects decay fastest. It treats time as administrative epochs and never asks whether the device's input representation is stable. Our nuisance correction is complementary to its feedback correction, and could in principle stack with it.

The signal-processing literature constrains the design and explains why phantom drift is real. Wang et al. show that the decimated discrete wavelet transform is time-variant, so its coefficients change with input shifts, most strongly in high-frequency bands. A redundant transform without downsampling is time-invariant. Chaudhury et al. give the dual-tree complex wavelet, whose magnitudes are approximately shift-invariant. These results tell us which device front-ends should exhibit phantom drift and which should not, which gives a falsifiable prediction. GeoECG shows that naive interpolation augmentation of ECGs creates physiologically impossible signals. It also shows that Wasserstein-geodesic augmentation reaches AUROC 0.931 on PTB-XL. This warns that "just make the device invariant in training" is not free.

Model Assessment and Selection under Temporal Distribution Shift adapts evaluation windows to change. It supplies a competing way to cope with non-stationarity, but it adapts to any change, including nuisance change, rather than ignoring it.

On the governance side, two lines of work set the regime. Advancing Science- and Evidence-based AI Policy argues for pre-committed if-then rules over static snapshots, and Luna et al. document how static rules decay. Our declared nuisance set is a small, concrete instance of a pre-committed rule that does not decay when pipelines change. We do not claim to model legislation; the legislative bottleneck analysis in Khan et al. is context, not a target.

Research Program

We first establish that phantom drift exists and measure its size. We train several ECG classifiers on PTB-XL, following the benchmark protocol of Strodthoff et al., with front-ends chosen to span shift sensitivity:

  • a strided ResNet;
  • a decimated-wavelet front-end;
  • a redundant-wavelet front-end;
  • a dual-tree complex wavelet front-end;
  • a ResNet trained with random-shift augmentation.

We then simulate post-market streams and inject purely nuisance changes into deployment. Nuisance scenarios are a change in the distribution of recording-window offsets, a 500→100 Hz→500 Hz resampling round trip (PTB-XL ships both rates), and mild time warping. Warping is not a group, so it tests how far the exact guarantee degrades when its assumption fails.

Second, we inject genuine decay. One scenario is a swap-out of the device for a weaker checkpoint, mirroring the language-model audit. Others are subgroup shift by age and sex using PTB-XL metadata, label-prevalence shift across the five superclasses, and added electrode noise.

Third, we run every monitor on every stream and record three things: the false alarm rate under nuisance-only change, the detection delay under genuine decay, and the query cost. The orbit test costs $|T|$ queries per record, so we also study subsampled orbits. A random subset of $T$ keeps invariance in expectation, and we check how much variance this adds.

Fourth, we compute the shift-sensitivity index per device. We test whether it predicts how badly raw monitors misfire, which is what would make it useful as a certification quantity.

Baselines

The naive reference point is a scheduled static audit. A fixed held-out batch is evaluated at the start and again after a set number of records, with a two-sample test on AUROC. This is what many certification regimes do today. It uses the same stream and the same total label budget as the continuous monitors. Static audits are known to be gamed by swap-outs, so beating this baseline is necessary but not impressive.

The strongest existing method for this question is the anytime-valid betting audit of Richter et al., applied to the device's raw per-record loss or output scores. It is the only cited continuous audit with a bounded false alarm rate, and it differs from ours only in what it bets on. The comparison is therefore nearly a controlled ablation. We run it with the same betting strategy, the same learned discriminator capacity where one is used, the same $\alpha$, and the same records. To equalize queries, we give it $|T|$ independent fresh records for each orbit we query, so it gets the same number of device calls. The reported reference numbers are detection up to 80% with false positive rate below 0.4 on language-model toxicity. On ECG nuisance shift, there is no reported result found on Lacuna for this test.

The second strong baseline is calibration-based CUSUM monitoring from Feng et al., which that paper found fastest among its options. We tune its thresholds on the same reference stream to the same nominal false alarm rate. For this method, there is no reported detection-delay number found on Lacuna. Its own evaluation used synthetic readmission simulations.

A third baseline is input-drift detection on shift-invariant dual-tree complex wavelet magnitudes. It catches timing-robust input changes but is blind to device swap-outs, since it never looks at the device. We also run the adaptive rolling window estimator of Han et al. on windowed loss, with alarms at a tuned deviation.

Two ablations isolate the key ingredient. The first applies orbit averaging with a fixed-sample test instead of betting, which separates the invariance from the sequential guarantee. The second runs the raw-loss betting test on the augmentation-trained device, which tells us whether device-side invariance alone already removes phantom drift.

Beating the strongest baseline means two things together. First, our false alarm rate stays at or below $\alpha$ under all group-structured nuisance scenarios while the raw betting test exceeds it. Second, our median detection delay under each genuine decay scenario is no more than modestly larger, which we define in advance as within 25%.

Why not just…?

Why not just require vendors to train with random shift augmentation? This is the real threat, and the augmented-device ablation is there to settle it. If augmented devices show no phantom drift under raw monitors, the audit's value shrinks to devices that were not trained that way. Two reasons suggest the threat is partial. A regulator audits whatever device it is given and cannot retrain it. And Wang et al. show that downsampling makes outputs depend on absolute position in a way that training-time averaging does not remove. We will report the fraction of devices and scenarios for which augmentation suffices, and say so plainly if it is most of them.

Why not just widen the alarm threshold? Widening trades false alarms for missed decay across all scenarios at once. Our test removes only the nuisance direction. The experiment that decides this is a power curve at a matched false alarm rate under nuisance shift. If threshold inflation matches our detection delay at equal false alarms, the proposal fails.

Why not just monitor calibration with CUSUM, or adapt the window? Calibration is computed from device outputs that themselves flicker under shift, so nothing stops it from misfiring under nuisance change. Adaptive windows, as in Han et al., respond to any change by design, nuisance included. Both are run as baselines on identical streams, so the result is directly observable.

Why not just detect input drift on shift-invariant features? It handles timing robustly but never queries the device. It therefore misses the most important regulatory failure, a vendor swap-out with unchanged inputs. The swap-out scenario tests this directly.

Decisive Checks

The idea is supported if three things hold. Under circular-offset and resampling nuisance changes, raw betting and CUSUM monitors on the decimated front-ends exceed nominal false alarm rates while the orbit test stays at $\alpha$. Detection delay for swap-outs and subgroup shifts stays within the pre-registered margin. And the shift-sensitivity index ranks devices by how badly raw monitors misfire, with time-invariant front-ends near zero, as the wavelet theory predicts.

The idea is refuted, or narrowed, under any of three outcomes. Phantom drift may be negligible on PTB-XL-scale offsets for every front-end. Augmentation alone may eliminate it. Or the orbit test may lose most of its power under realistic subsampled orbits. The time-warp scenario will tell us whether the guarantee extends past exact groups or whether the protocol must declare only shift sets that are true groups.