Methods that search for neural network designs automatically are almost always graded against one standard: how well the chosen network does once it is trained on a fully labeled dataset. The main users of label-free search are fields like clinical ECG analysis, where expert labels are scarce. A clinic fine-tuning a model on 200 annotated recordings needs the network that works best with 200 labels, and that may not be the network that works best with 20,000. This proposal asks whether the ranking of candidate architectures changes as the labeled set shrinks. It also asks whether a cheap signal can predict the ranking in the low-label setting without using labels. That signal is a few minutes of self-supervised pre-training per candidate, followed by simple checks on the resulting representations. Self-supervised pre-training means learning from unlabeled data by predicting parts of the signal from other parts.
The claim is concrete and can be tested. First, on 12-lead ECG, architecture rankings measured with 1% of the labels will disagree substantially with rankings measured with 100%. Second, "zero-cost proxies" are scores computed from one batch passed through an untrained network. Borrowed from vision, they will track the full-label ranking at best, and in sequence models they may track nothing more than model size. Third, a very short self-supervised run, scored without labels, will recover the low-label ranking at a small fraction of the cost of full pre-training. If the first part fails, meaning rankings are stable across label budgets, that is still a useful negative result. It would mean existing label-free search methods can be validated at full labels without loss, and the note is built so that this outcome gets reported.
What Is Known and What Is Missing
Three lines of work already remove labels from architecture search. Are Labels Necessary for Neural Architecture Search? ranks candidates by performance on tasks such as rotation prediction and jigsaw puzzles. It reports a rank correlation of about 0.8 with supervised accuracy, but it trains around 500 networks to get there. Neural Architecture Search With Random Labels trains a shared "supernet" on random labels and scores each sub-network by how far its weights rotate from initialization. A supernet is one large network whose weights are shared by all candidate sub-networks. Towards Self-supervised and Weight-preserving Neural Architecture Search searches with a contrastive objective and is the only one of the three that checks a low-label split, 10% of CIFAR-100. All three work on 2D images. All three validate their rankings mainly against full-label accuracy. These papers motivate the proposal: they show label-free search is viable, but they never ask whether the target they validate against is the right one for label-scarce users.
A second line cuts search cost by scoring untrained networks. Neural Architecture Search on ImageNet in Four GPU Hours (TE-NAS) scores trainability and expressivity at initialization, but its region-counting measure assumes ReLU activations. AZ-NAS combines four such scores and reports Kendall's τ of 0.710 on ImageNet16-120. EZNAS evolves a proxy formula that reaches τ = 0.65 on NAS-Bench-201. These methods supply the zero-cost baselines here. Two findings constrain the design. Training-free Neural Architecture Search for RNNs and Transformers found that on BERT-style models the best training-free proxy (τ = 0.27) lost to counting parameters (τ = 0.44). EZNAS notes that proxies separate good candidates from bad ones but lose resolution inside the top 10%. Any proposed estimator therefore has to beat parameter count, and it has to be judged on the top of the ranking, not only on overall correlation.
The third line is label-efficient ECG learning, which serves as the application setting. Self-supervised representation learning from 12-lead ECG data shows that contrastive predictive coding (CPC) is the stronger pre-training objective on ECG. CPC learns by predicting future stretches of the signal in a learned feature space. With CPC pre-training, an xresnet1d backbone reaches macro AUC 0.9418 on PTB-XL, against 0.9318 for supervised training. Benchmarking ECG FMs reports that pre-trained models are 3.3x to 9x more label-efficient. It also reports that a 3.8M-parameter state-space model, ECG-CPC, beats models up to 25 times larger in 5 of 7 task categories, and that ECG-JEPA does best below 1,000 labeled samples. A state-space model processes a sequence through a learned linear recurrence. That last detail matters for this proposal: the benchmark already hints that which architecture wins depends on how many labels are available. Both ECG papers use hand-designed backbones and trained each on different pre-training data, so architecture and data are confounded.
Core Claim and Estimator
The object of study is the regret of an architecture-selection rule when the downstream label fraction is $f$. Regret here is how far short the chosen network falls of the best available one:
$$ R_f(s) = \max_{a \in \mathcal{A}} \mathrm{AUC}_f(a) - \mathrm{AUC}_f(s^{*}), $$
where $\mathcal{A}$ is the candidate pool, $\mathrm{AUC}_f(a)$ is the test macro AUC of architecture $a$ after CPC pre-training and fine-tuning on a fraction $f$ of the labels, and $s^{*}$ is the architecture that scoring rule $s$ ranks first.
The proposed rule is called the short-horizon self-supervised probe. Each candidate is CPC-pre-trained for a fixed small budget, about 1–2% of the full pre-training steps, on unlabeled PTB-XL recordings. It is then scored using only unlabeled held-out data. The score combines two quantities. The first is the CPC future-prediction accuracy at the end of the short run, a direct measure of how quickly the architecture learns the pretext task. The second is the effective rank of the embedding covariance on held-out windows, which measures how many directions the representation actually uses. The combination weight is fitted once on a small separate set of architectures and then frozen.
The hypothesis behind this estimator is that low-label fine-tuning depends mainly on how well the pre-trained features already separate classes, and a short run should show which architectures start building such features. Full-label performance, by contrast, can recover from mediocre features through extended fine-tuning. The central ablation computes the same diagnostics on untrained networks, which turns the probe into a zero-cost proxy. That comparison isolates whether the short run itself is the useful ingredient.
Research Program
The first step is to build a small tabular benchmark for 1D ECG. The search space mixes the three families the ECG benchmark found competitive: 1D residual convolution stages in the style of xresnet1d, diagonal state-space layers, and attention layers. Per-stage choices cover width, depth, kernel size, downsampling position, and block type. Parameter budgets run from roughly 1M to 20M so that parameter count can be tested directly as a competitor. About 300 architectures are sampled. Each one is CPC-pre-trained with a fixed recipe on the training folds of PTB-XL with labels ignored, using 100 Hz signals, which Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL reports is sufficient. Each is then fine-tuned at label fractions of 1%, 5%, 10%, and 100%, drawn from the patient-stratified folds, with three seeds per low-label fraction because small-sample fine-tuning is noisy.
At 100 Hz and under 20M parameters, one full pre-training run is small. The total budget scales roughly as 300 architectures times one pre-training run plus about 10 fine-tuning runs each. This is the most expensive part of the project, but it is a one-time cost, and the benchmark will be released so that others can test selection rules on it.
The second step is analysis on the benchmark. It asks how Kendall's τ between the $f$-fraction ranking and the 100% ranking falls as $f$ decreases, and whether any disagreement is larger than the seed noise within each fraction. It asks which architectural features, such as state-space layers versus convolutions or depth versus width, gain or lose rank under scarcity. It also tests whether the pattern the ECG benchmark hinted at appears here too. That benchmark found attention-based models best below 1,000 samples, whereas the compact state-space model won overall.
The third step scores every selection rule against the $f$-specific ground truth. The measures are τ, top-10% τ, and regret $R_f$, each plotted against the GPU-hours the rule spends. External validity is checked by fine-tuning the top-ranked architectures on ICBEB2018, which the benchmark does not contain.
Baselines
The naive reference point is to do no search at all. This means using the xresnet1d backbone with the CPC recipe from Self-supervised representation learning from 12-lead ECG data, which reports macro AUC 0.9418 with full-label fine-tuning and 0.9318 for its supervised baseline. A second hand-designed reference is a state-space backbone matched to the ECG-CPC design, 3.8M parameters, as described in Benchmarking ECG FMs. Both are pre-trained and fine-tuned with exactly the benchmark recipe, so any gap reflects architecture alone. Reported low-label PTB-XL AUCs for a single backbone at 1% or 5%: no reported result found on Lacuna for these fractions, so both references will be measured here. A searched architecture counts as beating them only if its low-label AUC exceeds theirs by more than the spread across seeds.
The simplest competing selection rule is parameter count, restricted to the same compute budget. This baseline must be beaten because it was the strongest training-free rule on sequence models, with τ = 0.44 against 0.27 in Training-free Neural Architecture Search for RNNs and Transformers. FLOP count is reported alongside it.
The strongest existing zero-cost method is AZ-NAS, which reports τ = 0.710 on ImageNet16-120. It will be re-implemented for 1D inputs with its four components unchanged and computed on the same unlabeled batches. EZNAS-A and the NTK condition number from TE-NAS will also be computed, the latter only for ReLU candidates because of its activation assumption. These rules cost seconds per architecture. They are beaten only if the probe achieves lower regret at 1% and 5% labels, and the reported comparison will state clearly that the probe spends more compute to get there.
The strongest trained label-free method is the weight-angle metric from Neural Architecture Search With Random Labels. It will be applied to each candidate after a random-label run with the same step budget as the probe, so the two spend identical compute and differ only in objective and scoring.
The key ablation is the probe's own diagnostics computed at initialization, with zero steps of training. If that scores as well as the full probe, the short run adds nothing and the claim fails. A second ablation uses full-length CPC pre-training loss as the selection rule. It shows the ceiling for a pretext-task score and tests whether the short run's estimate is limited by its truncation.
Why not just…?
Why not just use the few labels you have to pick the architecture? This is the most serious threat. The alternative is to fine-tune each short-pre-trained candidate on the small labeled set and select by validation AUC on a held-out part of that set. It requires no new estimator, and at 5–10% labels it may work well. The expected weakness is at 1%, around 200 recordings for PTB-XL. There, most of the 71 statements have very few positive examples. That would make validation AUC noisy enough that selection mostly picks up seed noise, and splitting off a validation set costs scarce training labels. The benchmark answers this directly, because this rule is run at every fraction with equal compute, and its regret is measured. If it matches the probe everywhere, the recommendation becomes "use the labels," and the note will say so.
Why not just count parameters? On FlexiBERT, parameter count beat every proxy. The ECG benchmark, however, found a 3.8M model outperforming models 25 times larger, which suggests size is a poor guide in this domain. The benchmark deliberately spans a 20x parameter range to test this. Parameter count may still turn out to be a strong low-cost rule, and if so it will be reported as such.
Why not just apply AZ-NAS or another vision proxy? Those proxies were designed and validated on vision networks against full-label accuracy, and their resolution is known to weaken among top candidates. Nothing in the file shows how they behave on state-space layers or under low-label evaluation. Their 1D versions are baselines here, and the probe has to beat them on top-10% τ and on regret at low label fractions.
Why not just search on CIFAR and transfer, or skip search? Both Moving Beyond Handcrafted Architectures in Self-Supervised Learning, where transfer correlation drops to ρ = 0.23, and Towards Self-supervised and Weight-preserving Neural Architecture Search report that architecture rankings do not transfer well across domains. The no-search references above test whether search is worth doing at all in this domain.
Proposed Experiments and Decisive Checks
The main dataset is PTB-XL, with 21,837 records from 18,885 patients and patient-stratified folds, evaluated on the macro AUC label set of the cited CPC study. External transfer is tested on ICBEB2018. As a sanity check that the pipeline matches prior results, AZ-NAS will be re-run on NAS-Bench-201 to confirm its reported correlations.
Three results decide the claims. The first is whether Kendall's τ between the 1% and 100% rankings falls below 0.6 by a margin larger than the within-fraction seed variance. If it does not, the premise fails and the benchmark is released as evidence that full-label validation is adequate. The second is whether the probe, at no more than 2% of full pre-training compute per candidate, achieves lower 1% regret than parameter count, AZ-NAS, the weight-angle metric, and its own at-initialization ablation. The third is whether the probe's chosen architecture beats the xresnet1d and state-space references on ICBEB2018 at matched label counts. If the first result holds but the second fails, there is still a useful finding: current label-free search is optimizing for the wrong target, and fixing it is left open.
References
- Are Labels Necessary for Neural Architecture Search?
- Neural Architecture Search With Random Labels
- Towards Self-Supervised and Weight-preserving Neural Architecture Search
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
- AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Training-free Neural Architecture Search for RNNs and Transformers
- Self-supervised representation learning from 12-lead ECG data
- Benchmarking ECG FMs: A Reality Check Across Clinical Tasks
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
- Moving Beyond Handcrafted Architectures in Self-Supervised Learning