Choosing a neural network architecture usually means training many candidates and comparing them on labeled data. Cheap shortcuts exist: score untrained networks, or watch how quickly the training loss falls in the first few epochs. All of these shortcuts are judged on how well they predict fully supervised accuracy. Semi-supervised learning is the setting where a model gets a handful of labeled images, sometimes four per class, plus tens of thousands of unlabeled ones. There, the labeled data is too small to say much about which architecture is better. Any criterion computed from those few labels is mostly reporting our prior assumptions: how weights are initialized, how strongly they are regularized, and which handful of images happened to be labeled. This proposal asks whether architecture rankings for semi-supervised training differ from supervised ones. It also tests whether the training signal on the unlabeled data can stand in for the missing labels as a ranking criterion that is both cheap and stable when those prior choices change.
The concrete plan has three parts. First, we build a ground-truth table of semi-supervised accuracy for a stratified sample of NAS-Bench-201 architectures. Second, we measure how sensitive each candidate ranking criterion is to the prior choices above. Third, we test a new estimator, consistency training speed: a prequential (sequential-prediction) score summed over early epochs of FixMatch's unlabeled loss, built so that an architecture cannot game it by collapsing onto one class. If the supervised ranking already predicts the semi-supervised ranking, we will say so. Measuring that is part of the contribution, because nobody in the literature we found has done it.
Core Claim
We make two claims. The first is that, at 1–4 labels per class, criteria that score an architecture from its labeled-data likelihood rank it according to the prior rather than the data. These criteria include early training loss, Laplace-approximated evidence on the labeled set, and early validation accuracy on a tiny labeled split. Their rankings will reorder substantially when the initialization scale, the weight decay, or the identity of the labeled subset changes, even though those changes barely move the true semi-supervised accuracy ranking.
The second claim is that the speed at which an architecture drives down its unlabeled consistency loss carries the missing information, provided it is scored honestly. Two conditions make it honest. Samples whose confidence falls below the threshold must be charged maximal loss rather than ignored. Predictions that collapse toward a single class must be penalized. Scored this way, consistency speed should rank architectures for semi-supervised accuracy better than, and more stably than, label-based criteria and zero-cost proxies.
The second claim rests on an observation from Revisiting the Train Loss. That paper ranks architectures by summed early training losses (Training Speed Estimation, TSE) and justifies the score as a sequential, prequential estimate of marginal likelihood. Under scarce labels, the stream of labeled predictions behind that estimate is very short. We extend the stream to the unlabeled data, where FixMatch already generates targets. The risk in doing so is the one documented by The promises and pitfalls of deep kernel learning: when a network is fitted by maximizing its own evidence, it can "hack" the score by collapsing its feature space. An architecture that confidently predicts one class has a near-zero unlabeled loss. The collapse correction in our score exists to block that route, and we test whether it works.
Research Program
The method components come from FixMatch and from the search space and training protocol of NAS-Bench-201. FixMatch converts a weakly augmented view's prediction into a hard pseudo-label when its confidence exceeds $\tau = 0.95$, then trains the model to predict that label on a strongly augmented view. NAS-Bench-201 supplies 15,625 cells, small graphs of operations repeated through the network, along with their supervised accuracies.
Its supervised ranking is free to look up, which is why it is our most important competitor. For a architecture $A$ trained with FixMatch for $T$ early epochs, define the consistency training speed as
$$\mathrm{CTS}(A) = \frac{1}{T}\sum_{t=1}^{T}\left[\frac{1}{|U|}\sum_{x\in U}\Big(m_t(x)\,\ell_t(x) + \big(1-m_t(x)\big)\log C\Big) + \beta\,\mathrm{KL}\big(\bar p_t(A)\,\|\,\mathrm{Unif}_C\big)\right].$$
The symbols are as follows:
- $U$ is the unlabeled set and $C$ the number of classes.
- $m_t(x)$ is 1 if the weak-view confidence on $x$ passes the threshold at epoch $t$ and 0 otherwise.
- $\ell_t(x)$ is the cross-entropy of the strong view against the pseudo-label.
- $\log C$ is the loss of a uniform guess, charged to unconfident samples so that staying uncertain earns no credit.
- $\bar p_t(A)$ is the architecture's average predicted class distribution over $U$.
- $\beta$ weights the penalty for drifting toward a few classes; it is fixed once on a held-out dataset and not tuned per benchmark.
Lower scores rank higher. The score uses the same forward passes FixMatch already performs, so the cost is $T$ short epochs per candidate.
The prior-sensitivity analysis is what connects this to Bayesian model selection. The Bayesian Analysis of Complex, High-Dimensional Models and On the Geometry of Bayesian Graphical Models with Hidden Variables show that small data plus high dimension lets prior choices move posterior and evidence conclusions. BayesNAS shows this concretely for architecture search: its sparsity-intensity prior scale strongly shapes the final network. Those papers motivate us to treat the prior as an experimental variable.
For every criterion, we recompute the ranking under a fixed grid of perturbations: the initialization variance scaled by 0.5 and 2, weight decay scaled by 0.1 and 10, and five different draws of the labeled subset. We report two quantities. Stability is the Kendall rank correlation between rankings under different perturbations. Accuracy is the Spearman correlation and top-k regret against ground truth. The ground truth itself is computed under standard hyperparameters, so a good criterion must be both stable and accurate.
Baselines
The naive reference point is the architecture with the best supervised CIFAR accuracy in NAS-Bench-201, trained with FixMatch. It costs nothing and asks whether search for semi-supervised training is needed at all. For calibration, NAS-Bench-201 reports a best supervised CIFAR-10 test accuracy of 94.37% in its space. Alongside it we report random selection from the same candidate pool.
The strongest zero-cost competitor is the voting ensemble of synflow, jacob_cov, and snip from Zero-Cost Proxies for Lightweight NAS. It reports Spearman $\rho = 0.82$ against supervised accuracy on NAS-Bench-201, compared with 0.61 for a 20-epoch training proxy. It is strong here because synflow needs no labels and jacob_cov needs only inputs, so both apply directly to unlabeled batches. We also include the XGBoost surrogate over 13 proxies from NAS-Bench-Suite-Zero, fitted only on architectures outside the evaluation sample. That paper reports up to 42% better rank correlation than single proxies and documents a bias toward large models; we add its size-penalized variant.
The strongest training-based competitor is TSE from Revisiting the Train Loss, computed on the labeled cross-entropy inside the same FixMatch runs. That paper reports a 170–300% gain in one-shot rank correlation over validation accuracy in the supervised setting. The final competitor is early validation accuracy on a labeled validation split whose size matches the labeled training set, since a large labeled validation set would defeat the purpose of semi-supervised learning.
We do not know of any reported semi-supervised accuracy for NAS-Bench-201 cells; there is no reported result found on Lacuna for this. FixMatch's own numbers use a Wide ResNet backbone: 88.61% at 40 labels and 94.93% at 250 labels on CIFAR-10. We use them only as a sanity anchor for our training pipeline, not as a target for the cells.
Fairness rules apply to every comparison. All training-based criteria get the same $T$ epochs, batch size, augmentations, and seeds, and the same labeled subsets. Every hyperparameter, including $\beta$, the number of early epochs, and the proxy-ensemble weights, is chosen on CIFAR-100 and frozen before CIFAR-10 and SVHN are evaluated.
Two ablations isolate the key ingredient. The first removes the $\log C$ charge, giving FixMatch's usual masked loss, to test whether ignoring unconfident samples lets slow learners look fast. The second removes the collapse penalty, to test whether the deep-kernel-style hacking actually appears. A third comparison, CTS against labeled TSE within the same runs, isolates the contribution of the unlabeled stream.
CTS beats the competitors only if three things hold at 4 labels per class. Its mean Spearman correlation with ground truth must exceed the best competitor's by a margin larger than the across-seed spread. Its top-10 regret must be lower. Its stability under prior perturbation must be higher.
Why not just…?
Why not just take the best supervised architecture and train it with FixMatch? This is the most serious threat. If semi-supervised and supervised rankings agree, especially among the top 5% of architectures, then search for semi-supervised training is unnecessary. Our contribution would then shrink to showing that, plus the prior-sensitivity diagnosis. FixMatch's own observation gives some reason to doubt agreement: performance at 1–4 labels per class depends heavily on capacity and on which examples are labeled, which suggests that capacity and inductive bias interact with label scarcity. Our first experiment settles the question directly by measuring the supervised-to-semi-supervised rank correlation and the regret of the supervised pick, at 4 and at 25 labels per class.
Why not just run zero-cost proxies, which already need no labels? They are cheap and partly label-free, so they are a genuine competitor. Two known problems weigh against them. NAS-Bench-Suite-Zero shows they track parameter count, and Zero-Cost Proxies for Lightweight NAS reports that they rank poorly across varied topologies. More fundamentally, they score a network at initialization and so cannot see how it responds to strong augmentation during training, which is the mechanism FixMatch relies on. If the ensemble or the surrogate matches CTS in both accuracy and stability, we will recommend the proxies, since they cost three orders of magnitude less compute.
Why not just validate briefly on a small labeled split, or use labeled TSE? These are exactly the criteria our first claim says are dominated by the prior. They work in supervised search because thousands of labels outweigh the prior; with 40 labels they may not. The perturbation grid tests this directly. If their rankings stay stable when the labeled subset is redrawn, the first claim is wrong, and we will report it as wrong.
Why not keep the stochastic supernet as the model, as in NSA-id? Understanding and Exploring the Network with Stochastic Architectures trains a weight-sharing network over a fixed, uniform distribution of wirings and uses it as an ensemble for semi-supervised learning. It reports 2.75% CIFAR-10 error, from its ensemble result, and an expected calibration error of 0.0032. That approach answers a different question. It needs several stochastic forward passes at inference and architecture-specific parameters, and it never says which single network to deploy. We include it as a reference accuracy ceiling, not as a ranking competitor.
Proposed Experiments
We train a stratified sample of 300 NAS-Bench-201 cells, stratified by supervised accuracy and parameter count, with FixMatch on CIFAR-10 at 40 and 250 labels. The same cells are trained on CIFAR-100 at 400 labels, used only for tuning, and on SVHN at 40 labels, with three seeds and three labeled-subset draws each. Training uses a shortened schedule. Because short schedules can reorder rankings, the top 30 cells under each criterion are retrained at the full FixMatch schedule, and we report whether the conclusions survive. This table of results is itself a deliverable.
Three checks decide the outcome.
The first is the supervised-to-semi-supervised rank correlation and the top-10 regret of the supervised pick. A high correlation here means the strongest version of the claim fails.
The second is whether label-based criteria are measurably less stable under the prior grid than CTS. This decides the prior-sensitivity claim independently of whether CTS wins.
The third is whether removing the collapse penalty lets degenerate architectures, those whose average predicted class distribution has low entropy and whose final accuracy is poor, rise to the top of the uncorrected score. That would confirm the deep-kernel-style hacking appears in this setting and that the correction is necessary.
All three are reported together with compute measured in GPU-hours, so that a reader can see each criterion's accuracy for its cost.
References
- Revisiting the Train Loss: an Efficient Performance Estimator for Neural Architecture Search
- The promises and pitfalls of deep kernel learning
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- The Bayesian Analysis of Complex, High-Dimensional Models: Can It Be CODA?
- On the Geometry of Bayesian Graphical Models with Hidden Variables
- BayesNAS: A Bayesian Approach for Neural Architecture Search
- Zero-Cost Proxies for Lightweight NAS
- NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
- Understanding and Exploring the Network with Stochastic Architectures