Scale-Balanced Self-Supervision for Physiological Signals

When a model learns from unlabeled ECG or EEG by hiding part of the signal and reconstructing it, every time-step counts equally in the loss. In these signals, most of the energy and most of the duration belong to slow components such as baseline drift, respiration and delta waves. The clinically decisive events are short: a 40-millisecond QRS notch, an epileptic spike, a sleep spindle. As a result, the pretraining objective spends nearly all of its gradient on teaching the network to redraw slow waves, and very little on the brief shapes that later carry the diagnosis.

Segmentation research has already met and fixed this exact pathology in space. Large lesions dominate a voxel-wise loss and erase small ones, and a bounded size reweighting repairs it. This proposal carries that fix into the time domain and into the pretraining objective itself. A multi-scale temporal convolutional encoder is pretrained to reconstruct masked signal separately in each frequency band, with mask lengths matched to each band's natural timescale and bounded per-band loss weights. The bet is narrow and checkable. If scale bias in the pretext loss is a real cause of poor label efficiency, correcting it should help most on short, shape-dependent labels and on high-sampling-rate input. It should help little on rhythm labels, where the slow components already carry the answer.

Core Claim

The claim is that the scale composition of a self-supervised loss, not only the encoder architecture or the amount of data, decides which clinical labels can later be learned from few examples. Three published results motivate it. Optimizing Operating Points Using Lesion Size Reweighting shows that voxel-level cross-entropy lets large multiple-sclerosis lesions monopolize gradients. Its bounded exponential reweighting, parameterized by $(\alpha, \beta)$, fixes this, while strict inverse-size weighting is the worst method tested because it is numerically unstable. Benchmarking ECG FMs finds a "generalist gap": eight ECG foundation models, meaning large models pretrained on unlabeled ECG for reuse across tasks, struggle with fine morphological and structural targets. Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL reports that raising ECG sampling from 100 Hz to 500 Hz gives negligible gains for fixed-kernel CNNs. A single reading explains both of the last two observations: current training never makes the network pay for ignoring high-frequency detail, so extra resolution and fine morphology go unused.

Research Program

The encoder is a parallel multi-resolution 1D convolutional stack in the style of AttnSleep's two-branch network (AttnSleep) and TSception's banks (TSception). Following TSception, kernel lengths are set as fixed fractions of the sampling rate, so each branch covers the same span in milliseconds at 100 Hz and at 500 Hz. Both prior models were trained only with labels, and this work reuses their architecture without modification.

The pretraining targets come from a fixed, non-learned analysis filter bank. It splits each input $x$ into $K$ band signals $x_1, \dots, x_K$, running from the QRS and spike range down to the baseline-drift range. Targets are fixed so the encoder cannot make the task trivial by collapsing a band. Each band gets its own masking scale: band $k$ is hidden in spans whose length is proportional to that band's period. A short span hides a QRS complex but is filled in trivially at low frequency, while a long span hides respiratory drift but makes high-frequency content unpredictable. A shared decoder predicts every band, and the loss is

$$ \mathcal{L} = \sum_{k=1}^{K} w_k \, \frac{1}{|M_k|} \sum_{t \in M_k} \big(\hat{x}_k(t) - x_k(t)\big)^2, \qquad w_k = \frac{\min\!\big(\beta,\; \bar{E}_k^{-\alpha}\big)}{\sum_j \min\!\big(\beta,\; \bar{E}_j^{-\alpha}\big)}, $$

The symbols are as follows. $M_k$ is the set of masked time-steps for band $k$. $\hat{x}_k$ is the predicted band signal. $\bar{E}_k$ is the running mean energy of band $k$ over the pretraining corpus. $\alpha \in [0,1]$ controls how strongly low-energy bands are boosted, and $\beta$ caps the boost. Setting $\alpha = 0$ recovers equal band weighting. Leaving the weights uncapped ($\beta = \infty$, $\alpha = 1$) gives the unstable inverse weighting that the lesion-size work warns against. That gives the method two natural internal controls.

ECG-CPC is the CPC model in the ECG foundation-model benchmark; CPC (contrastive predictive coding) predicts summaries of upcoming signal blocks rather than raw samples. Because it is the strongest compact ECG model, the program also tests the same idea inside CPC. In that variant, each encoder branch forecasts its own future block, and the per-branch contrastive terms are combined with the same bounded weights. The point of this variant is to separate "scale balancing helps" from "masked reconstruction helps."

The analysis side of the program measures, during pretraining, the share of gradient norm coming from each band, with and without reweighting. It also probes the frozen features for QRS duration and ST-segment amplitude. The goal is to show that the mechanism moves where it should move.

Baselines

The naive reference point is the same multi-scale encoder trained from scratch on each label fraction. The strongest existing method is CPC pretraining of the xresnet1d backbone from Self-supervised representation learning from 12-lead ECG data. It is the method with published numbers on our main benchmark, reporting a fine-tuned PTB-XL macro AUC of 0.9418 against 0.9318 for supervised training and 0.927 under a linear probe. We also report the 0.939 macro AUC of CNN ensembles on the diagnostic task from Deep Learning for ECG Analysis. The ECG-CPC state-space model from Benchmarking ECG FMs is the second strong competitor: it is a 3.8M-parameter model that beats models 25 times larger in 5 of 7 task categories. No per-task number for ECG-CPC on PTB-XL was found on Lacuna, so we rerun it ourselves.

For sleep staging, the fully supervised reference is AttnSleep's 84.4% accuracy on Sleep-EDF-20.

Fairness is controlled at four points. All self-supervised methods see the same unlabeled pool, matched pretraining compute in GPU-hours, the same fine-tuning protocol, and the same hyperparameter search budget per method on the validation fold.

Two ablations isolate the key ingredient. The first trains the same multi-scale encoder with plain uniform masked reconstruction. It tests whether the kernel banks alone explain any gain. The second keeps band targets and scale-matched masks but sets $\alpha = 0$. It tests whether bounded weighting matters beyond decomposition. Beating the field means two things. At 1–10% of PTB-XL labels, macro AUC on form statements must exceed CPC-xresnet1d by more than the seed-to-seed spread across five seeds. At 100% labels, the model must reach at least 0.9418.

Why not just…?

Why not just pretrain the existing CPC model longer or on more data? CPC-xresnet1d is the real threat, and it is the baseline. Its encoder uses one receptive field per block, and its loss lives in latent space with no control over which timescales the latents encode. The CPC variant of our method answers this directly. If per-branch weighted CPC fails to beat plain CPC at matched compute, while masked scale balancing does not beat CPC at all, the claim fails and we report it.

Why not just whiten the spectrum of the input, or use a Fourier-magnitude loss? Whitening is a simple, plausible fix for the same imbalance, and we take it seriously. It is effectively the uncapped inverse-energy weighting that Lesion Size Reweighting found unstable, and it also amplifies electromyographic noise. A Fourier-magnitude loss discards timing within the window, while QRS shape is a time-localized feature. Both are run as baselines under the same encoder and budget. If whitened uniform reconstruction matches our method, the conclusion becomes "rebalance the spectrum," and we will state that rather than credit band-wise masking.

Why not just add multi-scale kernels and train with labels plus augmentation? AttnSleep, TSception and LGGNet already show that multi-scale kernels help when labels are plentiful. The open question is the low-label regime. There, Self-supervised representation learning from 12-lead ECG data finds supervised models trail self-supervised ones at every training size, with the widest gap on rare conditions. The from-scratch multi-scale baseline is run with jitter, scaling and time-warp augmentation tuned on validation, so this alternative is tested rather than assumed away.

Proposed Experiments

Pretraining for ECG uses all PTB-XL records with labels removed, at both 100 Hz and 500 Hz. Fine-tuning uses PTB-XL's recommended folds at 1%, 5%, 10%, 20% and 100% of training labels. We report macro AUC separately for diagnostic, form and rhythm statements, and for the rarest quartile of statements. Transfer is tested by fine-tuning on ICBEB2018 without further pretraining. For EEG, pretraining uses SHHS single-channel signal without labels and fine-tuning uses Sleep-EDF-20 at the same label fractions. Metrics are macro F1 and N1 F1, because N1 is the short, spindle-and-vertex-wave stage that AttnSleep singles out as hardest.

The decisive checks are pattern predictions, not just leaderboard numbers. First, gains over CPC should be largest on form statements and smallest on rhythm statements. If the gains are uniform, the improvement is generic regularization rather than scale correction. Second, the method should gain more from 500 Hz than from 100 Hz input, while baselines should reproduce the negligible-gain finding of Deep Learning for ECG Analysis. Third, frozen-feature probes for QRS duration should improve and probes for heart rate should not. Fourth, sweeping $\alpha$ should produce a peak at intermediate values, with instability or degradation near uncapped inverse weighting, mirroring the lesion-size result. If the first two checks fail, the hypothesis that pretext-loss scale bias limits label efficiency is wrong, whatever the aggregate AUC.