Pretraining Biosignal Encoders to Produce Independent Innovations

Self-supervised pretraining for heart and brain recordings usually asks an encoder to forecast its own future in latent space. The forecast is scored by a dot product (contrastive predictive coding) or by squared error. Both scores only check whether the mean of the future is right. They are blind to what statisticians call non-linear serial dependence: structure where the forecast error looks uncorrelated with the past, yet its size or shape still depends on the past. Volatility clustering is the textbook case. A beat-to-beat irregularity that grows and shrinks over seconds is a physiological example. We propose to pretrain encoders with a different target. After the model's best forecast, what is left over (the "innovation") should be statistically independent of everything the model has already seen. Independence is checked with distance covariance and kernel dependence measures, which detect any kind of relationship, not just linear ones.

The bet is concrete. Clinically relevant but label-scarce properties of ECG, such as rhythm irregularity and variability phenotypes, live partly in exactly the structure that mean-forecasting objectives push into the discarded error term. An encoder that is forced to absorb that structure into its state should need fewer labels to learn rhythm tasks. The same independence test, applied to a trained model's innovations, also gives a label-free way to choose among pretrained checkpoints. That matters when 1% labels leave too little data for honest validation.

Core Claim

Let $z_t$ be the encoder's latent state for signal window $t$. A small predictor maps $z_t$ to a forecast mean $\mu_t$ and a per-coordinate scale $\sigma_t$, and $e_{t+1} = (z_{t+1}-\mu_t)/\sigma_t$ is the standardized innovation. We train with

$$ \mathcal{L} = \underbrace{\mathcal{L}_{\text{NLL}}(z_{t+1};\mu_t,\sigma_t)}_{\text{heteroscedastic forecast}} + \lambda_v \mathcal{L}_{\text{var}}(z) + \lambda_c \mathcal{L}_{\text{cov}}(z) + \lambda_d \sum_{k=0}^{K} \widehat{D}\big(e_{t+1},\, z_{t-k}\big) + \lambda_r \mathcal{L}_{\text{rec}}. $$

Here $\mathcal{L}_{\text{var}}$ and $\mathcal{L}_{\text{cov}}$ are the variance hinge and off-diagonal covariance penalties from VICReg, used to prevent the trivial constant representation. $\widehat{D}$ is a batch estimate of a dependence measure that is zero only under independence. $K$ is the number of past lags checked. $\mathcal{L}_{\text{rec}}$ is a small reconstruction anchor that stops the encoder from discarding unpredictable content just to make innovations vanish.

The forecast term alone guarantees only that the error has zero conditional mean and correctly modeled variance given $z_t$. The dependence term additionally rules out higher-order and longer-lag dependence of the innovation on the past. To make that term small, the encoder must make $z_t$ a sufficient state. The new element is this dependence term. We test whether it moves label efficiency beyond what a well-specified mean-and-variance forecaster already achieves.

Relation to Prior Work

Several papers supply the dependence machinery but apply it along a different axis. dCovTS defines the auto-distance covariance function and a multi-lag portmanteau test. These detect serial dependence such as GARCH-type volatility and non-linear autoregression, which autocorrelation and Ljung–Box tests miss. We reuse its estimator, both as the training penalty and as the diagnostic. Kernel Constrained Covariance supplies the guarantee that universal kernels make the population measure zero exactly under independence. Its no-free-lunch theorem also constrains us: no finite-batch penalty can catch every alternative, so we report which dependence types the penalty actually removes rather than claiming full independence. Ranks of distances offers a monotone-invariant variant that we test as a way to avoid bandwidth tuning. Bandwidth sensitivity is a documented weakness of kernel tests.

The closest existing work brings kernel dependence into self-supervision, but always as dependence across views or across feature coordinates, never between forecast error and past. SSL-HSIC maximizes the Hilbert-Schmidt Independence Criterion between representations and view identity on static images. Variance Covariance Regularization Enforces Pairwise Independence shows that decorrelating a BatchNorm-ReLU projector's output implicitly minimizes this criterion between encoder coordinates. Graph VICRegHSIC makes that coordinate penalty explicit for graphs. Functional maximal correlation and its EEG adaptation HFMCA maximize correlation between views under orthonormality constraints. These motivate the proposal because they show dependence objectives can replace negative sampling. HFMCA in particular documents false negatives in contrastive biosignal learning. None of them treats time as a sequence of states whose innovations should be independent.

Two temporal methods set the bar. Dynamics Contrastive Learning adds a learnable forward model before InfoNCE and proves identifiability of non-linear latent dynamics under Gaussian noise. We keep the explicit forward model but drop negatives, and we target non-Gaussian, heteroscedastic noise, where their assumptions fail. PULSE splits a physiological signal into shared system parameters and window-specific initial conditions. Our approach does not impose that split. It asks only that whatever the state carries leaves independent residuals.

On the motivating side, Benchmarking ECG FMs reports that foundation models struggle with hidden phenotypes and out-of-distribution targets. That is the failure we expect independent innovations to address.

Research Program

We first build the method on synthetic signals where the ground truth is known. These include oscillators whose noise amplitude follows a GARCH process, the Lorenz system observed through heteroscedastic noise, and a non-Markov variant where the future depends on a lag the encoder must learn to retain. On these we measure two things. One is latent recovery $R^2$, following the protocol of Dynamics Contrastive Learning. The other is whether the trained model's innovations pass the dCovTS portmanteau test. This stage tells us whether the penalty removes dependence that a heteroscedastic forecaster leaves behind, and how batch size and lag count $K$ trade off against test power.

Second, we pretrain on 12-lead ECG. We use the same 3.8M-parameter structured state-space backbone that Benchmarking ECG FMs found most label-efficient. Keeping the architecture fixed means the comparison isolates the objective.

For the dependence estimator we compare three options: exact distance covariance at batch size 256, which is quadratic but affordable at that size; random-Fourier-feature HSIC as in SSL-HSIC, which is linear in batch size; and rank-of-distance statistics. Their wall-clock cost is reported against the Randomized Dependence Coefficient cost figures. That paper notes that exact HSIC and distance correlation take over ten minutes at 100,000 samples, so we only ever compute them per batch.

Third, we turn the test into a tool. For each pretraining run we save checkpoints and compute the innovation portmanteau statistic on held-out unlabeled data. We then ask whether it ranks checkpoints by downstream 1%-label performance better than the pretraining loss does.

Baselines

The naive reference point is supervised training from scratch with the same backbone at each label fraction. With full labels, Self-supervised representation learning from 12-lead ECG data reports a purely supervised macro AUC of 0.9318 on PTB-XL. Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL reports about 0.939 for a supervised CNN ensemble on diagnostic statements.

The strongest existing competitor is contrastive predictive coding on a state-space backbone (ECG-CPC). Benchmarking ECG FMs reports it to be 3.3 to 9 times more label-efficient than supervised models. On PTB-XL, CPC reaches a finetuned macro AUC of 0.9418 and a frozen linear-probe macro AUC of 0.927 with full labels (Self-supervised representation learning from 12-lead ECG data). We found no reported result on Lacuna for CPC on PTB-XL at 1%, 5% or 10% label fractions, so we rerun it ourselves.

On MIT-BIH, the competitor is PULSE, which reports 84.77% accuracy with 1% labels and 87.41% linear-probe accuracy (PULSE). We also include VICReg on augmented windows and SSL-HSIC adapted to temporal crops, representing static dependence-based self-supervision.

Every method gets the same backbone and the same unlabeled pretraining data, the PTB-XL training records with labels stripped. Each also gets the same number of pretraining steps and an equal grid of 20 hyperparameter configurations. Selection uses the same fixed labeled validation subset at each fraction, with five label-subset seeds.

Two ablations isolate the key ingredient. The first removes the dependence term, keeping the heteroscedastic forecast plus VICReg terms. The second further replaces the heteroscedastic forecast with plain squared error. Beating CPC means a higher macro AUC averaged over seeds at 1% and 5% labels, with non-overlapping seed intervals. Beating the first ablation means the same, concentrated on the PTB-XL rhythm statements, where our hypothesis predicts the gain.

Why not just…?

Why not just use CPC with a state-space model, which already wins? It may win at full labels, and we expect our method to match rather than beat it there. The case against it is narrower. Its InfoNCE denominator treats other windows as negatives, even though in ECG they often share the same pathology. HFMCA documents this contamination. In addition, Dynamics Contrastive Learning shows symmetric contrastive learning fails at system identification under low noise, with 41.0% Lorenz $R^2$ against 94.1% for its forward-model variant. If CPC equals our method at 1% labels on rhythm statements, the hypothesis is wrong for ECG.

Why not just use VICReg with a non-linear projector, which already enforces kernel independence? The independence that Variance Covariance Regularization proves is between coordinates of a representation. Ours is between forecast error and past states. A representation can have perfectly independent coordinates while its dynamics leave volatility in the residual. The VICReg baseline and the synthetic GARCH test directly measure whether coordinate independence implies innovation independence.

Why not just add a variance head to the forecaster, which captures volatility? This is the real threat, which is why the forecast term in our objective is already heteroscedastic and the first ablation is exactly this alternative. A Gaussian mean-and-variance forecaster removes second-moment dependence on the current state. It does not remove dependence on older lags or skewed, higher-order structure. If the ablation matches the full method on ECG, then real ECG innovations hold little beyond second-moment structure. The training contribution would then collapse, and we would report only the label-free selection result.

Why not just use masked reconstruction? Masked autoencoders work at scale, as in Brant. However, PULSE reports masking-based and other self-supervised baselines at 57–67% on MIT-BIH with 1% labels, against 84.77% for its dynamical approach. We include a masked-reconstruction baseline on the same backbone to confirm this on PTB-XL.

Proposed Experiments and Decisive Checks

On PTB-XL we report macro AUC on all statements, and separately on diagnostic, form and rhythm statements. We do this at 1%, 5%, 10% and 100% labels, under both frozen linear probing and finetuning, with patient-stratified splits. On MIT-BIH we report accuracy at 1% labels and under linear probing against PULSE. On synthetic systems we report latent $R^2$ against Dynamics Contrastive Learning's 94.1% Lorenz figure, along with innovation test p-values.

Three results decide the proposal. First, on GARCH-modulated synthetic data the full method should pass the innovation independence test and the heteroscedastic ablation should fail it. If both pass or both fail, the penalty adds nothing measurable. Second, on PTB-XL at 1% and 5% labels, the gain over the heteroscedastic ablation should appear mainly on rhythm statements. A uniform gain would suggest generic regularization rather than captured serial structure. Third, across checkpoints, the innovation portmanteau statistic should correlate more strongly with 1%-label AUC than the pretraining loss does, measured by Spearman correlation over at least 30 checkpoints. If it does, practitioners gain a selection rule that spends no labels, even if the training gains turn out modest.