Deep networks for medical signals such as the 12-lead electrocardiogram (ECG) are limited in two ways. Expert labels are scarce. The network shapes themselves are chosen by hand, usually copied from vision or language models. A recent benchmark of eight ECG foundation models (large networks pretrained on unlabeled data, then adapted to many tasks) found that the architecture matters more than size. A hand-built 3.8M-parameter state-space model, ECG-CPC, beat models 25 times larger in 5 of 7 clinical task categories (Benchmarking ECG FMs: A Reality Check Across Clinical Tasks). A state-space model is a sequence layer that carries a compact running memory of the signal. The architecture was still picked by a person. Automated architecture search, known as neural architecture search (NAS), could pick it instead. But the usual search methods rank candidate networks by accuracy on labeled validation data, which is exactly what is scarce here.
We propose to run the architecture search using the same unlabeled pretraining task that works best for ECG: contrastive predictive coding (CPC). CPC trains a network to predict a summary of the next few seconds of signal from the past, and to tell the true future apart from futures drawn from other recordings. The search space mixes convolution, state-space, and attention operations, and no label or pretrained teacher network is used. The question is not whether search can beat a generic ResNet. The question is whether a network found with a label-free forecasting criterion yields better label-efficiency curves than the strongest hand-designed ECG backbone at matched parameters and compute.
Core Claim
Our claim is that, for continuous physiological signals, a CPC forecasting loss can rank architectures in agreement with downstream label efficiency. This holds only if the forecasting horizon is long enough that a network cannot win by copying local waveform shape. Under that condition, the loss can drive a teacher-free search whose result beats hand-designed ECG-CPC when only 1–10% of labels are available.
Three prior lines of work bear on this claim.
Two methods already combine search with self-supervision, and both constrain our design. Self-supervised Neural Architecture Search (SSNAS) replaced the label loss in DARTS with a SimCLR contrastive loss on 2D image cells. SimCLR learns by making two augmented copies of one image agree. On label-scarce CIFAR-100, SSNAS reached 83.36% accuracy, while supervised DARTS collapsed to 64.46%. Masked Distillation Advances Self-Supervised Transformer Architecture Search (MaskTAS) found that training a weight-sharing transformer supernet on raw masked reconstruction diverges. It fixed this with a frozen pretrained masked-autoencoder teacher. We differ from both. We use temporal forecasting rather than augmentation-based instance discrimination, because Self-supervised representation learning from 12-lead ECG data showed CPC beats SimCLR, BYOL, and SwAV on ECG. We use no teacher, because no strong one exists for 12-lead ECG. And we search over state-space and convolution hybrids rather than image cells or vision transformers.
The medical search papers motivate the gap without closing it. Heart-Darts and Scalable Neural Architecture Search for 3D Medical Image Segmentation (SCNAS) are fully supervised. Evolutionary Neural Architecture Search with Dual Contrastive Learning (DCL-ENAS) makes the architecture predictor label-efficient but still trains every candidate with labels, and it tested only on single-lead PhysioNet 2017. We borrow methods from these papers: the Gumbel-softmax trick of SCNAS, which samples only a few operations per connection, and the pairwise-ranking predictor of DCL-ENAS as a fallback.
The search objective is simple. For a candidate architecture $a$ with weights $w$, the encoder turns an ECG into a sequence of latent vectors $z_t$. A recurrent summary $c_t$ is formed from past latents. The CPC loss at horizon $k$ scores the true future latent $z_{t+k}$ against $N$ negatives $z_j$ drawn from other recordings:
$$ \mathcal{L}_{\text{CPC}}(a,w) = -\,\mathbb{E}_{t}\sum_{k=k_{\min}}^{k_{\max}} \log \frac{\exp(z_{t+k}^\top W_k c_t)}{\sum_{j} \exp(z_j^\top W_k c_t)} . $$
Here $W_k$ is a learned projection for horizon $k$. The architecture weights are updated to minimize this loss on held-out unlabeled recordings, and the network weights are updated on training recordings. The quantity we treat as the key ingredient is the minimum horizon $k_{\min}$. Our hypothesis is that short horizons reward architectures with large local receptive fields that interpolate waveform detail, while horizons of one or more cardiac cycles reward architectures that model rhythm and cross-lead structure. The latter is what the clinical labels depend on.
Research Program
The search space is a fixed macro-skeleton: a strided convolutional stem, then a stack of cells, then CPC context and prediction heads. Each cell edge chooses among dilated 1D convolutions of several widths, an S4-style state-space layer, local windowed attention, cross-lead mixing, identity, and zero. Stage widths and depths are also searchable. The space deliberately contains ECG-CPC's own design, so the hand-designed baseline is one point in the space, and "search found nothing better" is a possible outcome.
The search follows SCNAS-style stochastic relaxation, sampling two operations per edge to bound memory. A penalty caps the number of parameter-free skip operations per cell, targeting the degenerate collapse documented in SSNAS.
Because MaskTAS showed that label-free supernet training can diverge, the first work package is diagnostic rather than a search run. We train about 60 architectures sampled from the space standalone with CPC. Standalone means each network is trained on its own, without shared weights. We then fine-tune each at 1% and 10% of PTB-XL labels. This gives ground truth for two measurements. The first is how well the final CPC loss of each standalone network ranks downstream macro AUC, as a function of $k_{\min}$. The second is how well the weight-sharing supernet's inherited CPC loss reproduces that ranking. If sharing weights destroys the ranking, the search falls back to regularized evolution with a DCL-ENAS-style pairwise predictor trained on CPC losses of standalone runs. That fallback is still label-free.
The discovered architecture is then pretrained at full budget and evaluated across label fractions. We also analyze what was found, for example where state-space layers versus convolutions are placed and whether cross-lead mixing appears early. We use Centered Kernel Alignment (CKA) to compare its representations against ECG-CPC, since the benchmark reported that models with similar scores learn divergent internal features.
Pretraining uses the ECG corpora in the public datasets of the ECG foundation-model benchmark with labels stripped. PTB-XL test folds are strictly excluded.
Baselines
The naive reference point is supervised training from scratch of a standard xresnet1d on the same labeled fraction. For full PTB-XL diagnostic labels, Self-supervised representation learning from 12-lead ECG data reports 0.9318 macro AUC. The CNN ensembles of Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL reach about 0.939. We rerun both at every label fraction.
The strongest existing method is hand-designed ECG-CPC, the state-space CPC model from the ECG foundation-model benchmark, together with the CPC-pretrained xresnet1d of Mehari and Strodthoff. The latter reports 0.9418 macro AUC fine-tuned and 0.927 with a frozen linear probe on PTB-XL diagnostic statements. We found no reported per-dataset AUC for ECG-CPC in the file (no reported result found on Lacuna for ECG-CPC on PTB-XL). We therefore retrain it ourselves with the same corpus, the same pretraining epochs, and the same fine-tuning recipe. For fairness, it receives a hyperparameter budget (depth, width, learning rate, horizon) equal in GPU-hours to our entire search. Beating it means a higher mean macro AUC across patient-stratified folds at 1%, 5%, and 10% labels, at equal or lower parameter count, with confidence intervals that do not overlap the tuned baseline. Parity at 100% labels is acceptable.
A random-search baseline samples architectures from the same space, pretrains each with CPC, and selects by CPC loss under the same compute. This isolates whether the search method adds anything beyond the search space.
An SSNAS port swaps CPC for SimCLR with time-series augmentations inside the identical search. This isolates the forecasting objective.
The horizon ablation reruns our search with $k_{\min}=1$ step versus $k_{\min}$ of about one cardiac cycle. This isolates the key ingredient.
A supervised-DARTS baseline in the Heart-Darts style searches on the 1% labeled split. It tests the known failure mode on ECG.
Why not just…?
Why not just scale or tune the hand-designed ECG-CPC model? This is the real threat. The benchmark already shows that a small, well-chosen state-space model dominates. It is plausible that careful tuning of depth, width, and horizon recovers whatever the search finds. We therefore give the tuned baseline our full search budget. We report the result honestly if the discovered architecture falls inside the tuned baseline's confidence band. In that case the paper's contribution becomes the ranking study: evidence about which label-free criterion predicts label efficiency.
Why not just run supervised architecture search on the few labels available? SSNAS reports supervised DARTS collapsing from 83.36% to 64.46% under label scarcity on CIFAR-100. Heart-Darts and SCNAS never tested scarce labels. Our supervised-DARTS baseline at 1% PTB-XL labels checks whether the collapse carries over to ECG rather than assuming it.
Why not just search the downstream head on a frozen foundation model? Neural Architecture Search for Sentence Classification with BERT gained 3.0% on 500-sample GLUE by searching heads, which is cheap and competitive. But the same work found that frozen backbones were consistently suboptimal, which suggests that the backbone limits performance. We include a head-search baseline on frozen ECG-CPC with equal compute. If it matches our result, backbone search is not worth the cost.
Why not just random-search the space? Random search is a notoriously strong NAS baseline. The matched random-search arm settles this directly.
Proposed Experiments
The decisive first check is the ranking study. We compute Kendall-τ between the CPC loss of about 60 standalone-trained architectures and their 1%-label PTB-XL macro AUC, for short versus long $k_{\min}$. DCL-ENAS treats τ above 0.6 as useful for ranking. If CPC loss does not correlate with downstream AUC at either horizon, then a label-free search criterion is unsound for ECG, and we stop and report that negative result.
If the ranking holds, the second check is the label-efficiency curve on PTB-XL. We compare macro AUC at 1%, 5%, 10%, and 100% labels under 10-fold patient-stratified cross-validation, against all baselines above.
The third check concerns the benchmark's "generalist gap." We evaluate frozen and fine-tuned transfer on the ECG foundation-model benchmark's categories. We focus on Cardiac Structure & Function and Acute Care Prediction, where fixed foundation models struggle. The prediction is that the gains from forecasting-driven search concentrate in rhythm-dependent and structure-dependent tasks rather than in adult interpretation. A uniform small gain everywhere would instead suggest that the search found a generically better network rather than one matched to cardiac structure.
Search cost is reported in GPU-hours alongside the Heart-Darts 16-hour reference, so the practical price of removing the hand-design step is explicit.
References
- Benchmarking ECG FMs: A Reality Check Across Clinical Tasks
- Self-supervised Neural Architecture Search
- Masked Distillation Advances Self-Supervised Transformer Architecture Search
- Self-supervised representation learning from 12-lead ECG data
- Heart-Darts: Classification of Heartbeats Using Differentiable Architecture Search
- Scalable Neural Architecture Search for 3D Medical Image Segmentation
- Evolutionary neural architecture search with dual contrastive learning
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
- Neural Architecture Search for Sentence Classification with BERT