Reusing a Pretrained Encoder's Own Forecast Errors to Adapt Biosignal Models Under Shift

Self-supervised ECG encoders are pretrained by predicting their own future. Contrastive Predictive Coding (CPC) is one such method: it trains an encoder on unlabeled recordings to guess the internal representation of the next few hundred milliseconds of signal from what came before. After pretraining, the part that makes these guesses is thrown away, and the encoder is used as a fixed feature extractor for diagnosis. We think this discards a useful signal. At deployment, the encoder can still make its guesses and check them against the signal that actually arrives. No labels are needed for this check. When the incoming data differ from what the model was pretrained on, for example a new hospital, a noisier recording device, or a patient population it rarely saw, those errors should grow in characteristic ways. That makes them a label-free, per-recording alarm and correction signal.

We propose to keep the pretext predictor alive at inference time. Its prediction errors feed a small recurrent "error memory," a running summary of recent surprises. This summary rescales and shifts the frozen features before a lightweight diagnostic head reads them. Everything runs in the forward pass, with no gradient steps at test time. The error memory is trained only on the small labeled set used for fine-tuning. The bet is that a model trained this way degrades less under distribution shift than linear probing, full fine-tuning, or gradient-based test-time training. We also expect it to be much cheaper than the last of these.

Core Claim

A CPC-pretrained encoder carries a built-in shift detector: its latent forecast error has a known reference distribution, computed on the pretraining data. Routing a normalized version of that error through a forward-pass memory lets a label-efficient classifier recalibrate itself on shifted recordings. Two things would be new. First, the adaptation signal comes from the pretext task itself rather than from ground-truth outcomes. TAFAS and PETSA, by contrast, need observed future targets. Second, the adaptation is recurrent state rather than test-time backpropagation. TAFAS, PETSA, and TTT++ all use backpropagation at test time.

Concretely, let $c_t$ be the encoder's context vector at step $t$, $z_{t+k}$ the encoding of the signal $k$ steps later, and $W_k$ the pretrained CPC prediction head. The residual we reuse is

$$ r_t = \Sigma_k^{-1/2}\big(z_{t+k} - W_k c_t - \mu_k\big), $$

where $\mu_k$ and $\Sigma_k$ are the mean and covariance of that residual on the pretraining corpus. A value of $r_t$ near zero therefore means "as surprising as usual." We add a scalar per-step contrastive surprise, the InfoNCE loss of the true future against negatives. The error memory is a gated recurrence,

$$ m_t = \lambda \odot m_{t-1} + (1-\lambda)\odot g(r_t), $$

in which $g$ is a small network, $\lambda$ is a vector of learned decay rates spanning fast and slow timescales, and $m_t$ produces a per-channel scale and offset applied to the frozen features. The decay spectrum borrows from the hierarchical decay layers of Multi-Recurrent Networks for Traffic Forecasting. The split between a predictor and a separate residual-memory block borrows from VARNN.

Where This Sits

Three lines of prior work motivate the proposal.

The first is on ECG foundation models. Self-supervised representation learning from 12-lead ECG data compared CPC against instance-discrimination methods (SimCLR, BYOL, SwAV) on PTB-XL. CPC won: fine-tuned CPC reached 0.9418 macro AUC and frozen linear evaluation reached 0.927. Benchmarking ECG FMs evaluated eight foundation models across 26 clinical tasks. It found that a 3.8M-parameter state-space model trained with CPC wins most task categories and that foundation models are 3.3x to 9x more label-efficient than supervised training. The same benchmark notes that evaluation is almost entirely in-distribution. Together these tell us that CPC is the right backbone and that shift is the untested regime.

The second line closes the loop with runtime errors. Recurrent Networks and NARMA Modeling feeds the previous prediction error back into a recurrent network. VARNN keeps a dedicated residual memory. MoML (MoML) meta-learns adapters that are updated from streaming errors. All of these require the true target $y_t$ to arrive at runtime. That is reasonable for forecasting, but it rules out diagnosis, where the label never arrives. Our change is to substitute the pretext residual, which is always observable, for the task residual.

The third line is test-time adaptation. TAFAS (Battling the Non-stationarity in Time Series Forecasting via Test-time Adaptation) calibrates gated modules around a frozen Chronos model using partially observed ground truth. PETSA (Accurate Parameter-Efficient Test-Time Adaptation for Time Series Forecasting) does the same with low-rank adapters. Both methods name gradient latency at test time as a bottleneck. TTT++ constrains the design space. It shows that updating weights on a self-supervised proxy loss under severe shift distorts features and hurts the main task, and that matching feature moments to stored source statistics fixes much of this. Our normalization by pretraining statistics $(\mu_k, \Sigma_k)$ is the analogue of that anchoring. The difference is that we never let the proxy loss move the encoder's weights. We expect this to avoid the distraction failure by construction, and we test that expectation directly.

Research Program

We will pretrain, or reuse, a CPC encoder on unlabeled ECG, keeping the prediction heads $W_k$ and recording residual statistics on the pretraining data. On top of the frozen encoder we train only the error-memory module and a diagnostic head, using 1%, 10%, and 100% of PTB-XL labels.

One risk is that residuals are nearly constant on clean in-distribution training data. The memory would then learn to ignore them. We will counter this by applying label-preserving perturbations to training recordings, such as baseline wander, lead gain changes, additive noise, and resampling. These make residual variation co-occur with feature corruption during training. Whether this augmentation is necessary is itself one of the ablations.

We will analyze three things. First, whether the residual magnitude alone predicts accuracy loss on shifted cohorts, which tests it as a detector. Second, whether the memory's modulation moves shifted features back toward source statistics, which tests it as a corrector. Third, how the gains depend on memory timescale for 10-second records versus long continuous streams.

A second application tests transfer to intracranial EEG. There we attach the same module to a masked-reconstruction model in the style of Brant, using masked-patch reconstruction error in place of the CPC residual. The Brant pretraining corpus comes from only nine subjects, which makes cross-subject shift the natural stress test.

Baselines

The naive reference is a linear probe on the frozen CPC encoder with no adaptation. In-distribution on PTB-XL, it scores 0.927 macro AUC per Self-supervised representation learning from 12-lead ECG data. The strongest non-adaptive competitor is full fine-tuning of the CPC encoder, which the same paper reports at 0.9418 macro AUC on PTB-XL. We will also train a supervised S4 model from scratch, because Benchmarking ECG FMs found such models surprisingly competitive.

The strongest adaptive competitor is TTT++-style adaptation. It takes gradient steps on the CPC loss at test time with feature-moment alignment, following TTT++. This is the gradient-based way of using exactly the same label-free signal, so it is the direct rival to our design. We found no reported result on Lacuna for TTT++ or any test-time adaptation method on ECG. The only reported TTT++ numbers are in vision, for example 9.8% average error on CIFAR-10-C, so we will run it ourselves. We will also include feature-statistic re-normalization alone, a gradient-free moment-matching baseline.

For fairness, every method uses the same frozen or initialized encoder, the same labeled subsets, and the same augmentation pool. Each method gets the same budget of hyperparameter trials, tuned only on an in-distribution validation split. No method sees shifted data during tuning.

Our key ablations remove individual ingredients. We replace the residual input with zeros. We replace it with the raw pair of predicted and actual latents, which isolates normalization against pretraining statistics. We remove the perturbation augmentation. We collapse the memory to a single timescale.

Beating the baselines means two things. The method must lose clearly less macro AUC than the fine-tuned model on shifted splits while staying within noise of it in-distribution. It must also match or exceed TTT++-style adaptation at a fraction of its inference latency.

Why not just…?

Why not just give a recurrent probe the full latent sequence? The residual is a deterministic function of the encoder's outputs, so a big enough head over the latent sequence could in principle learn it. This is the most serious threat to the idea. We settle it with a parameter-matched gated recurrent probe over the same latents. If that probe matches our module on shifted splits, the contribution reduces to "use sequence heads," and we will report it that way. Our hypothesis is that the probe, trained on limited labels, cannot rediscover the pretext predictor $W_k$ or its pretraining-calibrated statistics. Those statistics are the one piece of information about "normal" that labels do not supply.

Why not just fine-tune the encoder on the labeled data? Fine-tuning gives the best in-distribution number (0.9418 AUC). But Benchmarking ECG FMs explicitly flags cross-cohort shift as untested. Nothing in fine-tuning reacts to a new recording's statistics. Our shifted-split results will show whether fine-tuning's in-distribution advantage survives.

Why not just run test-time training on the CPC loss? It uses the same signal. However, TTT++ shows that unconstrained proxy updates degrade the main task under severe shift: vanilla TTT-R reaches 14.3% error on CIFAR-10-C against 9.8% with alignment. Both TAFAS and PETSA also report test-time gradient cost as a bottleneck. We report accuracy and wall-clock latency side by side. If gradient-based adaptation is both more accurate and tolerable in cost, our advantage narrows to latency, and we will say so.

Why not just re-normalize features to source statistics? This is cheap and gradient-free. It captures part of what our method does. It cannot distinguish a noisy recording from a genuine pathology, because both shift feature moments. The residual may be able to tell them apart, since a genuine pathology can still be predictable from its own past. We test this directly on pathology-rich subsets.

Proposed Experiments

All shifts are built from PTB-XL metadata described in PTB-XL. We train on the high-quality recordings (77% of the dataset) and test on the records with baseline drift and static noise. We train on middle-aged adults and test on the youngest and oldest age bands. We train on one sex and test on the other. We train at 500 Hz and test on 100 Hz signals upsampled. We measure macro AUC on superclasses and subclasses at 1%, 10%, and 100% label fractions. Where licensing allows, we also run cross-dataset transfer to the other public ECG datasets pooled in Benchmarking ECG FMs.

The decisive check has three parts. The method must shrink the gap between in-distribution and shifted macro AUC relative to fine-tuned CPC. It must beat the parameter-matched recurrent probe. The zero-residual ablation must lose that gain. If any of the three fails, the core claim fails.

Secondary checks cover three things: the correlation between mean residual magnitude and per-cohort AUC drop, inference latency against TTT++-style adaptation, and leave-one-subject-out seizure detection by $F_2$ score with a Brant-style backbone. For the streaming case, we run one-step heart-rate regression on BIDMC HR, where VARNN reports a roughly 44% error reduction over dynamic baselines. There we test whether pretext residuals recover part of that benefit without access to the true target.