Self-supervised pretraining on unlabeled electrocardiograms (ECGs) already lets small models match fully supervised accuracy with a fraction of the labels. The strongest current recipe asks only the final layer of the encoder to predict what comes next in the signal. We propose spreading that prediction task across the encoder's depth, one frequency band at a time. First, a wavelet transform splits each ECG into bands, from fast, sharp detail (the spike of each heartbeat) down to slow trends (the ST segment level, baseline drift). Then each intermediate layer is trained to predict the future content of one assigned band, at a look-ahead distance that matches that band's time scale. Shallow layers predict fine detail a few milliseconds ahead. Deep layers predict slow structure seconds ahead.
The bet is that the last-layer-only objective lets the encoder ignore information that is cheap to discard but diagnostically important. The clearest evidence is that current ECG foundation models do well on standard rhythm labels but fail on "hidden" targets such as structural heart abnormalities and long-term outcomes. Giving each layer its own band-specific forecasting job should produce features that keep both scales available. That should matter most when downstream labels are scarce. The extra heads are thrown away after pretraining, so the deployed model has the same size and speed as the baseline.
Background and the gap
Three lines of work each touch two of the three ingredients.
Label-efficient ECG learning. Self-supervised representation learning from 12-lead ECG data compared contrastive predictive coding (CPC) against methods that learn by telling augmented copies of the same record apart from other records (SimCLR, BYOL, SwAV). CPC trains an encoder to pick the true future latent out of a set of distractors. It won on PTB-XL. Benchmarking ECG FMs: A Reality Check Across Clinical Tasks found that a 3.8M-parameter state-space model trained with CPC (ECG-CPC) beat models up to 25 times larger in 5 of 7 task categories, with a 3.3x to 9x label-efficiency gain. A state-space model is a sequence model that carries a running compressed memory of the signal. These two papers motivate our starting point: CPC on a compact encoder. They also expose the gap we target, the weak performance on hidden phenotypes. Both apply their loss only at the top of the encoder.
Wavelets in time-series models. A wavelet transform splits a signal into bands that stay localized in time. Existing work uses wavelets in three places:
- inside the architecture, as in WaveSSM, which builds state-space memory from wavelet frames and reports 0.942 AUROC on PTB-XL versus 0.934 for S4;
- as trainable filters, as in mWDN;
- as a loss on the final output, as in Beyond Time.
All three train with labels, and none supervises intermediate layers. We reuse their decomposition machinery but place it in the pretraining targets, not in the architecture or the output loss.
Deep supervision with frequency or scale structure. T-MLP attaches output heads to every layer of a coordinate network so that early layers fit coarse shape and later layers fit residual detail. FreqExit aligns wavelet detail coefficients of early-exit layers with a full-model teacher. DeCon applies contrastive losses at several decoder depths and finds gains under few labels. These supply the method: attaching losses at multiple depths. FreqExit also supplies a constraint. It reports that heavily weighted high-frequency auxiliary losses disrupt low-frequency learning, which we must control for.
None of these papers learns from unlabeled sequences with band-specific forecasting targets at intermediate depths. T-MLP fits one known function. FreqExit distills from a supervised teacher. DeCon has no frequency structure.
Core Claim
Assigning wavelet bands to encoder depths, each with a forecasting horizon matched to its time scale, gives better label efficiency than two things:
- the same bands all supervised at the final layer;
- deep supervision with band-agnostic targets.
The advantage should be largest on the hidden-phenotype tasks where terminal CPC falls short.
Concretely, the pretraining objective is
$$ \mathcal{L} = \mathcal{L}_{\mathrm{CPC}} + \sum_{l \in S} \lambda \, \mathrm{InfoNCE}\big(g_l(z_{l,t}),\; e_{j(l)}(c_{j(l),\,t+h_{j(l)}})\big), $$
where the terms are defined as follows:
- $\mathcal{L}_{\mathrm{CPC}}$ is the standard terminal CPC loss.
- $S$ is a set of supervised intermediate layers.
- $z_{l,t}$ is layer $l$'s hidden state at time $t$.
- $g_l$ is a small prediction head.
- $j(l)$ is the wavelet level assigned to layer $l$.
- $c_{j,t+h_j}$ is the block of level-$j$ discrete wavelet coefficients in the window starting $h_j$ steps ahead.
- $e_j$ is a learned linear embedding of that block.
- InfoNCE contrasts the true future block against blocks from other times and records.
- $\lambda$ is a single shared weight.
The horizon is set as $h_j \propto 2^j$. This follows from the dyadic structure of the discrete wavelet transform: level-$j$ coefficients are spaced $2^j$ samples apart. A fixed number of coefficient steps ahead therefore corresponds to a look-ahead that doubles with each level.
Because the targets are fixed wavelet coefficients of the real signal, the heads cannot collapse to a trivial solution. Forecasting in coefficient space also avoids a known problem with masked reconstruction on biosignals: random patch masking creates artificial step edges at patch boundaries, and missing patches can often be filled in by simple interpolation of slow trends.
Our main hypothesis is that bands should go from fine to coarse with depth, because receptive field and useful look-ahead both grow with depth. This is the opposite ordering from T-MLP's coarse-first design for coordinate fitting. We treat the ordering as something to test, not something to assume.
Research Program
Encoder. We build the method on the encoder that is strongest in the file: an S4-style state-space encoder of ECG-CPC's size. We add an xresnet1d variant to connect to the Mehari–Strodthoff results. We use a Daubechies-4 decomposition to five levels at 100 Hz; the file reports no diagnostic gain from 500 Hz on PTB-XL. The five levels map onto five evenly spaced depths of the encoder.
Pretraining corpora. We use two:
- A controlled corpus: the unlabeled PTB-XL training folds, split by patient.
- A larger corpus: the public datasets used in the ECG foundation-model benchmark, with every evaluation test split removed.
Analyses. Three analyses accompany the headline numbers:
- Layer-wise linear probes that decode each wavelet band and each diagnostic label from each layer. These show whether band information is actually stratified by depth and kept, or overwritten.
- A loss-weight sensitivity sweep, to test the FreqExit concern about interfering losses.
- A fine-tuning study comparing three readouts: last layer only, a learned mixture of all layers, and freezing the lower half. Features stored at different depths may need a multi-layer readout to be useful.
Baselines
Supervised from scratch. The first reference point is the same encoder trained only on the available labels. On the full PTB-XL label set, Self-supervised representation learning from 12-lead ECG data reports 0.9318 macro AUC for the best purely supervised model. Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL reports about 0.939 for CNN ensembles.
Terminal CPC. The strongest existing method is terminal CPC on the same backbone. Self-supervised representation learning from 12-lead ECG data reports:
- 0.9418 macro AUC fine-tuned;
- 0.927 with a frozen encoder and linear readout.
ECG-CPC in the benchmark paper is the same idea on a state-space model. The file gives its 3.3x–9x label-efficiency factor but no per-fraction PTB-XL numbers. I found no reported result on Lacuna for PTB-XL macro AUC at 1%, 5%, or 10% labels, so we will reproduce these curves ourselves.
WaveSSM. WaveSSM (0.942 AUROC supervised) is the strongest architecture that already uses wavelets. We run it both supervised and with terminal CPC.
Ablations. Four ablations isolate the key ingredient:
- (A) terminal multi-band: all five band-forecasting heads attached to the last layer.
- (B) deep CPC: standard CPC heads at the same intermediate depths, with no band targets.
- (C) shuffled assignment: bands mapped to depths at random.
- (D) reversed assignment: coarse bands at shallow layers.
Fairness controls. Every method uses:
- identical pretraining data;
- matched pretraining GPU-hours;
- the same encoder size at deployment;
- the same fine-tuning protocol and patient-level folds.
Each method gets the same tuning budget of eight configurations, chosen by validation-fold AUC.
What counts as beating. The claim holds if the method does three things together:
- beats terminal CPC by more than the standard deviation across seeds (five seeds) at 1% and 10% labels on PTB-XL subclass and 71-statement tasks;
- beats ablations A and B;
- does not lose at 100% labels.
A win only over supervised training would not count.
Why not just…?
Why not just put a wavelet or frequency loss at the output, as Beyond Time does? This is the most serious threat, and ablation A is designed to test it. Beyond Time shows terminal frequency losses help supervised forecasting. Targeted Activation Penalties shows that terminal constraints can let unwanted features reappear in deeper layers, whereas constraining intermediate layers directly suppresses them. That is suggestive but was measured in a different setting. If ablation A matches the full method, the contribution reduces to a better pretraining target, and we would report it that way.
Why not just deep-supervise every layer with ordinary CPC? DeCon shows band-agnostic multi-level contrastive losses already help in low-label dense prediction. Ablation B tests whether that alone explains any gain. Ablations C and D test whether the specific pairing of band to depth matters. If shuffled assignment does as well as the ordered one, the wavelet stratification is not the operative ingredient.
Why not just feed wavelet bands as input channels, or use WaveSSM with terminal CPC? Fixed wavelet features with a shallow network lag well behind end-to-end CNNs on PTB-XL (Deep Learning for ECG Analysis). Fixed filters also underperform trainable ones across 40 UCR datasets (mWDN). Architectural wavelets are a different matter: WaveSSM plus CPC is a real competitor and is run as a full baseline. Our method is orthogonal to it and can also be applied to WaveSSM; we will report that combination.
Why not just pretrain a bigger terminal-CPC model longer? The benchmark paper found a 3.8M-parameter CPC model beating models 25 times larger. Scale is therefore not the obvious fix for the hidden-phenotype gap. We match compute, not parameters, so extra training time cannot explain any gain.
Proposed Experiments and Decisive Checks
Main experiment: PTB-XL. We fine-tune and linearly probe at 1%, 5%, 10%, 30%, and 100% of labels. We evaluate at the superclass, subclass, and 71-statement levels, reporting macro AUC. We also report accuracy on rare classes separately, since terminal CPC's largest gains were on minority classes.
Hidden-phenotype test: benchmark categories. On the multi-dataset benchmark, we focus on the structural and outcome categories where foundation models are weakest. The pre-registered prediction is that the gap over terminal CPC is larger on these categories than on rhythm statements. If the gain is uniform or concentrated on rhythm, the "missing multi-scale biomarker" explanation is wrong, even if the method helps.
Mechanistic check: layer-wise probes. We decode band-$j$ coefficients from layer $l$ with linear probes. We expect band-selective decodability to peak at the assigned depth. We also expect the final layer to still decode fine detail better under our method than under terminal CPC.
Transfer check: outside ECG. We rerun the label-ratio sweep on HAR and MIT-BIH, following Self-Supervised Learning for Time Series: Contrastive or Generative?, which reports that MAE gains 5.53% F1 over no pretraining at 1% labels on HAR. This tests whether scale-matched forecasting helps on non-periodic sensor data, where MAE currently wins at low label fractions.
Failure criterion. The claim fails if ablation A or B comes within seed noise of the full method at 1% and 10% labels.
References
- Self-supervised representation learning from 12-lead ECG data
- Benchmarking ECG FMs: A Reality Check Across Clinical Tasks
- WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention
- Multilevel Wavelet Decomposition Network for Interpretable Time Series Analysis
- OLMA: One Loss for More Accurate Time Series Forecasting
- T-MLP: Tailed Multi-Layer Perceptron for Level-of-Detail Signal Representation
- FreqExit: Enabling Early-Exit Inference for Visual Autoregressive Models via Frequency-Aware Guidance
- Beyond the Encoder: Joint Encoder-Decoder Contrastive Pre-Training Improves Dense Prediction
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
- Targeted Activation Penalties Help CNNs Ignore Spurious Signals
- Self-Supervised Learning for Time Series: Contrastive or Generative?