When chemists or agronomists train a one-dimensional convolutional network on spectra, GPU time is not what makes architecture choice expensive. A spectrum is one curve of a few hundred to a few thousand intensity values along the wavelength axis, measured by Raman, infrared or near-infrared instruments. A small 1D convolutional network, which slides learned filters along that axis, trains on such data in seconds to minutes. The scarce resource is the laboratory measurement. Each labeled sample needs a wet-chemistry assay, so cohorts of a few dozen are common; one Raman fish-composition study had 39 specimens. Unlabeled spectra, by contrast, are cheap: the instrument produces them for free. When every candidate architecture is scored by cross-validation on those few labels, each comparison spends statistical budget. With many candidates, the winner is partly the one that happened to fit the validation folds.
We propose to search 1D spectral architectures using only unlabeled spectra. Each candidate network is scored by how well it learns a self-supervised surrogate task on unlabeled spectra from the same instrument: reconstructing masked wavelength bands from their neighbors. The score is read from gradient statistics at initialization plus a very short training run on that task. The labeled samples are touched once, to train the chosen architecture. The claim is that, below roughly a hundred labels, this choice beats label-based cross-validated search. It also beats existing training-free scores, which mostly measure network size. We test the claim on a new benchmark that measures each architecture's true small-sample quality.
Core Claim
Existing low-cost architecture search treats compute as the cost. On small spectral datasets the real cost is selection overfitting. A useful score for this regime has two properties. It must be label-free, so it cannot overfit the few labels. It must also depend on the data, so it reflects the wavelength structure of the instrument rather than the network's size. Prior work shows that current training-free scores often have neither property. HyTAS benchmarks 12 such scores on 2,000 vision-transformer models for hyperspectral images. It finds that many scores work equally well when fed random noise instead of real data, and that they favor larger models. That paper treats the noise result as a puzzle. We treat it as the design target: make the score depend on real spectra through a surrogate task, and verify the dependence by an explicit noise-input ablation.
Why This Gap Is Real
Three findings motivate the proposal.
First, the depth limits of spectral data clash with the size bias of existing scores. Hyperspectral image classification using deep convolutional neural networks found that per-pixel accuracy on Pavia peaked at two convolutional layers (87.54%) and dropped when a third layer was added.
Second, existing scores reward size. Training-free scores, also called zero-cost proxies, rate an untrained network from a forward or backward pass. ZiCo scores a network by the mean-to-spread ratio of its gradients across a few minibatches. It shows that earlier scores barely beat simply counting parameters (Kendall τ 0.54 for Synflow versus 0.57 for parameter count on NATS-Bench). HyTAS reports the same size bias on hyperspectral images.
Third, standard search spaces miss the operations spectra need. NAS-Bench-360 shows that architecture rankings from vision tasks carry almost no information on scientific tasks. On 1D sequences, a search space with large kernels mattered more than the search algorithm. This matches the 64-wide filters that FishCNN needed to span multi-peak chemical bands.
Existing spectral architecture search does not cover the label-scarce 1D case. A2S-NAS and STAS train a "super-network" containing every candidate operation with shared weights, using 3D convolutions over image cubes. Both use labels, and STAS targets image restoration. HyTAS is the closest work: its ZiCo++ adds layer-wise decay to counter size bias and reaches Spearman ρ = 0.73 on Indian Pines. It covers only transformers on image patches, and its ground truth assumes thousands of labeled pixels. HyTAS also reports a random-forest predictor reaching about 0.8 rank correlation from 20 to 50 trained models. That predictor still uses labels to train those models, and it is the kind of label consumption we want to avoid when only 40 samples exist. Finally, SPARCS uses "spectral" in the eigenvalue sense, for fully connected layers. It is unrelated to spectroscopy, and we use it only as a pruning comparison.
Research Program
We first build a 1D spectral search space shaped by the domain literature. It varies depth from one to five convolutional stages, kernel size from 7 to 63, dilation, stride or pooling along the wavelength axis, and channel width. It also includes an optional attention head after the convolutions, as in the LSTM-CNN-attention soil model. We sample about 500 architectures from this space.
The surrogate task works as follows. Contiguous wavelength windows are masked in unlabeled spectra, after the standard SNV correction (each spectrum rescaled to zero mean and unit variance) and Savitzky-Golay smoothing. The network's convolutional trunk must reconstruct the masked windows. This task rewards filters whose reach and frequency response match real peak widths and shapes, which a noise input cannot reward.
For each architecture $a$, the score is
$$S(a) = Z_{\text{surr}}(a) - \beta\, L_{\text{surr}}^{(k)}(a),$$
where $Z_{\text{surr}}$ is the ZiCo statistic, with HyTAS-style layer decay, computed on the surrogate loss. $L_{\text{surr}}^{(k)}$ is the surrogate validation loss after $k$ training steps, with $k$ around 50. $\beta$ is a single weight fixed once on a held-out dataset, never tuned per task. The short-run term borrows the low-fidelity idea of budgeted performance estimation: train briefly as a stand-in for full training, checking that the budget preserves rankings. We apply it to the surrogate task instead of to labels.
The central artifact is a ground-truth benchmark. Because these networks are small, we can afford to train every architecture at label budgets $N \in \{40, 100, 400, \text{full}\}$, over 20 random label draws each. Test performance is measured on a fixed held-out set. This gives each architecture's expected small-sample quality, which is the quantity a practitioner actually cares about. No current benchmark records it, because all of them assume abundant labels.
Baselines
The naive reference points are partial least squares regression and a fixed two-layer 1D CNN. Partial least squares is the chemometric default: it projects spectra onto a few latent directions and regresses on them. The two-layer CNN mirrors the Pavia result. Both are trained on the same $N$ labels with the same preprocessing, and their few settings are tuned by inner cross-validation. Beating them means a higher mean test $R^2$ (or overall accuracy for classification) across label draws at $N \le 100$.
The strongest competitor is label-based search: nested cross-validation with random search over the same 500-architecture space, using the same $N$ labels. Full training is cheap here, so we give it equal or greater GPU time than our method. This is the method a careful chemometrician would actually run. Beating it means two things over 20 draws: lower regret, meaning the gap between the true small-sample quality of the chosen architecture and that of the best architecture in the space; and a smaller gap between cross-validated and test performance.
The strongest training-free baselines are ZiCo++, the parameter count, and ZiCo itself. Each ranks the same space with the same minibatch budget. Beating them means higher Spearman correlation with the ground-truth small-sample quality, together with lower regret.
For full-data context, the cited LUCAS hybrid model reports test $R^2$ = 0.949 for organic carbon on an 85/15 split. That is a large-label number. We reproduce the model as a fixed-architecture reference rather than a regime-matched target.
Two ablations isolate the key ingredient. The first computes the same score on random-noise inputs, which tests whether the score uses the data at all. The second computes the same score on the labeled samples with the true target, which tests whether avoiding labels is what helps.
Why not just…?
Why not just run cross-validated random search, since training is cheap? This is the real threat. At $N = 400$ and above we expect it to win or tie. Our claim is specific to $N \le 100$, where choosing the best of 500 noisy validation estimates is known to inflate those estimates. The benchmark measures this inflation directly as the cross-validation-to-test gap. If label-based search still has lower regret at $N = 40$, the core claim fails, and we will report that.
Why not just use ZiCo++ or parameter count off the shelf? HyTAS shows these scores are partly insensitive to data and biased toward size. The Pavia depth result suggests that size bias selects wrong architectures for spectra. The noise-input ablation and the correlation comparison test this on 1D CNNs, where no reported result exists.
Why not pretrain one fixed CNN on the unlabeled spectra and skip search? This uses the same unlabeled data along a different route, so it is a serious alternative. We run pretrained-fixed against pretrained-selected. If architecture choice adds nothing once pretraining is in place, the contribution narrows to "self-supervised pretraining suffices", and we would say so.
Why not stay with partial least squares? On 39 fish samples, FishCNN reached a cross-validated $R^2$ of 0.811, against 0.752 for the best classical model. Gains of this kind justify architecture choice at all, but they come from single studies. We keep partial least squares in every comparison.
Proposed Experiments
The primary testbed is LUCAS topsoil Vis-NIR spectra, with organic carbon, nitrogen, CaCO₃ and pH as targets. We simulate label budgets by drawing small labeled sets and treat the remaining spectra as an unlabeled pool. The second testbed is per-pixel spectra from Indian Pines and Pavia, treated as 1D curves. There, unlabeled pixels are genuinely abundant, and the published 1D-ConvCapsNet (99.18% overall accuracy on Indian Pines) and Pavia CNN results give orientation points. The fish Raman data serve as a real 39-sample case study, though they are too small to build a ground truth. On that dataset, a chosen architecture can be compared against FishCNN's 0.811 only under the same leave-out protocol.
No reported result exists on Lacuna for zero-cost proxies ranking 1D spectral CNNs, so the training-free baselines' correlations will be measured fresh.
The decisive checks are three. First, regret at $N = 40$ and $N = 100$ must be lower than that of label-based search across all four LUCAS targets. Second, the score must lose most of its correlation with ground-truth quality when fed noise. Third, the chosen architectures should not drift deeper than the true best architectures, unlike those chosen by parameter count. If the first check fails while the second holds, the conclusion becomes a narrower one: the score understands spectra, but labels remain the better guide.
References
- HyTAS: A Hyperspectral Image Transformer Architecture Search Benchmark and Analysis
- Hyperspectral image classification using deep convolutional neural networks
- ZiCo: Zero-shot NAS via inverse Coefficient of Variation on Gradients
- NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks
- Machine Learning for Raman Spectroscopy-based Cyber-Marine Fish Biochemical Composition Analysis
- A2S-NAS: Asymmetric Spectral-Spatial Neural Architecture Search for Hyperspectral Image Classification
- Spectrum-Aware and Transferable Architecture Search for Hyperspectral Image Restoration
- Spectral Architecture Search for Neural Networks
- Spectral Data-Driven Prediction of Soil Properties Using LSTM-CNN-Attention Model
- Rethinking Performance Estimation in Neural Architecture Search