Automated architecture search (NAS, the practice of letting an algorithm pick a network's layers and connections) is expensive because every candidate design must be trained before anyone knows whether it is good. Unsupervised filter learning offers a different way to fill a network with weights. In this older line of work, convolution filters are computed directly from the statistics of small image patches, for example as the leading principal components of patch covariance. That computation takes an eigendecomposition rather than thousands of gradient steps. This note proposes using it as the inner loop of architecture search. Each candidate network has its filters filled in layer by layer from patch statistics, a linear classifier is fit on its output, and that classifier's validation accuracy serves as the candidate's fitness score. Training by backpropagation happens only for the final winner.
The question that matters is not whether such analytic networks are competitive as final models; they are not on hard data. The question is whether they rank architectures the way trained networks do, and whether that ranking stays faithful in two settings where supervised search is known to break: small labeled sets and non-image signals. If they do, search becomes cheap and adapts to the data it is run on, since filters come from that modality's own statistics rather than a vision-tuned menu. If they do not, that is a clean negative result about how far patch statistics reach.
Core Claim
For a candidate architecture $\alpha$, define the analytic fitness
$$F(\alpha) = \mathrm{Acc}_{val}\big(\mathrm{ridge}\big(\Phi_\alpha(X_{tr})\big)\big),$$
where $\Phi_\alpha$ is the network $\alpha$ with every convolution's filters set, layer by layer and front to back, from the patch statistics of the previous layer's outputs on unlabeled training inputs $X_{tr}$. Ridge regression is then fit on the pooled features with the labels, and $\mathrm{Acc}_{val}$ is held-out accuracy. The claim has three parts. First, the rank correlation between $F(\alpha)$ and the fully trained accuracy of $\alpha$ is substantially higher than that of the same pipeline with random filters. Second, it is at least as high as a short-SGD proxy given equal wall-clock time. Third, the advantage widens when labels are scarce or the input is not a natural image.
The novelty sits in a gap between two literatures, and each half of that gap is filled by existing work. PCANet (PCANet) computes cascaded PCA filters in closed form and trained on 100,000 images in 30 minutes against 6 hours for a ConvNet. Convolutional Kernel Networks (CKNs) learn per-layer filters without labels by approximating a Gaussian patch kernel. Both supply the method we reuse, but both fix a hand-chosen two-stage pipeline. NAS systems take the opposite approach: DARTS (DARTS), regularized and large-scale evolution (Real et al.), search-space evolution (NSE), and LLM-driven search (AgentNAS) all automate topology but obtain every candidate's weights by supervised training.
The closest intersecting work does not close the gap. arch2vec (arch2vec) is unsupervised, but what it learns without labels is an embedding of the architecture graph, not the filters; each queried architecture is still trained. SSF-CNN (SSF-CNN) uses K-SVD patch dictionaries inside a fixed ResNet and never searches topology. We use unsupervised filters for something neither line has tried: as the evaluation step of search.
Research Program
The first component is the analytic builder. For each convolution node, we extract patches at that node's kernel size from its input feature maps and center them. The node's filters are the top-$c$ eigenvectors of the patch covariance, where $c$ is the node's channel count. A variant uses spherical k-means centroids on whitened patches, following Coates et al. and Roth et al.. Channels are normalized to unit variance in place of batch normalization, and a ReLU follows each convolution, the nonlinearity CKNs show arises from the kernel approximation.
Skip connections and pooling need no fitting. Where branches merge, the summed map is the input statistic for the next node, so the builder respects the graph's order of evaluation. The final features are average-pooled or second-order pooled, following PCANet-II, whose covariance pooling avoids PCANet's $2^L$ histogram explosion. Ridge regression is then fit in closed form. Patch statistics are estimated from a fixed subsample, and we will report how ranking quality depends on its size.
The second component is the search itself. Regularized evolution and random search are run with $F$ as fitness. The top few candidates are then fully trained, with the winner chosen on validation. We will also test whether initializing that final training from the analytic filters speeds it up, though this is secondary.
The third component is analysis. We will identify where the proxy fails by sorting architectures on which $F$ and trained accuracy disagree. One possible cause is the clutter issue noted for PCANet, where the highest-variance patch directions are not the discriminative ones. Another is depth, since analytic features may degrade in deep cells, which would favor shallow designs. We will report the proxy's correlation with parameter count and depth to check that it is not simply a size detector.
Baselines
The naive reference point is random search with a matched budget, together with a hand-designed Wide ResNet tuned the way NAS-Bench-360 tunes its Wide ResNet baseline. Any search method must beat both to be worth running.
The ablation that isolates our key ingredient is the identical pipeline with random orthogonal filters in place of patch-derived ones, at the same widths and with the same ridge readout. If patch statistics matter, analytic filters must beat this random-filter proxy in rank correlation. We will treat a Kendall tau gap smaller than the spread across random seeds as failure.
The second ablation swaps PCA filters for whitened k-means and adds second-order pooling. This tells us which unsupervised learner carries the signal.
The strongest cheap-evaluation competitor is a short-SGD proxy: each candidate is trained by backpropagation for whatever number of steps fits the same per-candidate wall-clock time as the analytic build, on the same data subsample and hardware. The proxy beats it if its rank correlation is equal or higher at equal cost.
The strongest end-to-end baseline on NAS-Bench-201 is arch2vec, which reports 94.18% on CIFAR-10, 73.37% on CIFAR-100, and 46.34% on ImageNet-16-120 for its best-found cell (arch2vec). We will compare using the benchmark's lookup tables and report the number of architecture evaluations and the wall-clock time used. Beating it means matching its final test accuracy within benchmark noise at lower compute, not necessarily exceeding it.
On NAS-Bench-360, the strongest reported system is AgentNAS, best on 11 of 17 tasks, with 38.71% error on Spherical and 13.06% average error on Unseen NAS against 15.04% for conventional NAS (AgentNAS). The file has no reported per-task Wide ResNet numbers for NAS-Bench-360 (no reported result found on Lacuna for those values), so we will rerun the benchmark's Wide ResNet ourselves. Because AgentNAS uses an LLM-proposed search space, we compare under its search space where it is reproducible. Otherwise we compare on the DARTS-style space, and we state clearly which comparison is which.
Why not just…?
Why not just use short supervised training as the proxy? This is the most serious threat. Early-stopped accuracy is a well-understood ranking signal and needs no new machinery. The equal-wall-clock comparison above settles it directly. Our expectation that analytic filters win is strongest in the small-label regime: SSF-CNN found that freezing patch-derived filter structure gave 70.42% Rank-1 on IIITD Newborn against roughly 60% for fine-tuning, which suggests gradient-trained weights are noisy when labels are few. If short-SGD matches us everywhere, including at small label budgets, the proposal's main claim fails, and we will report it that way.
Why not just use random filters, since random features already give a linear readout? If architecture alone determines ranking, random filters would be as good and cheaper. That is exactly why this is our central ablation rather than a side note. Evidence that data-derived filters carry real information comes from PCANet and CKNs: CKNs reach 82.18% on CIFAR-10 and 0.39% error on MNIST with unsupervised filters and a linear classifier (CKNs). But accuracy of the final model is not the same as ranking fidelity, so only the experiment can answer this.
Why not just use a tuned Wide ResNet or fine-tune a pretrained backbone? NAS-Bench-360 reports that a tuned Wide ResNet often beats existing NAS methods on non-vision tasks. That is a real bar, and we include it rather than dismiss it. Pretrained image backbones carry edge priors with no obvious meaning for physical or signal data such as fluid simulations, muscle-activity recordings, or genomic sequences. Our method computes filters from the target modality itself. If the search does not beat the tuned Wide ResNet on a majority of NAS-Bench-360 tasks, its practical value is limited to cheaper search in image domains.
Why not just use the graph-embedding approach of arch2vec? arch2vec cuts the number of architectures that must be evaluated, while we cut the cost of each evaluation. The two are complementary, so we will also run arch2vec's Bayesian optimization with $F$ as its objective.
Proposed Experiments and Decisive Checks
The first experiment is ranking fidelity on NAS-Bench-201. We compute $F$ for a stratified sample of several thousand cells on CIFAR-10, CIFAR-100, and ImageNet-16-120. We then measure Kendall tau against the benchmark's trained accuracies, both over the whole space and among the top 5% of cells, since search quality depends on how well the proxy separates good cells from each other. This is compared against the random-filter and short-SGD proxies. The same measurements are repeated with 1%, 10%, and 100% of training labels available to the ridge readout, while patch statistics always use all unlabeled images. The decisive pattern is a tau advantage over short-SGD that grows as the label fraction shrinks.
The second experiment is end-to-end search: on NAS-Bench-201 against arch2vec and random search at matched compute, and on NAS-Bench-360, where Darcy Flow, NinaPro, Spherical, PSICOV, and ECG test cross-domain behavior. On NAS-Bench-360 we will also measure how well candidate rankings computed on CIFAR-100 transfer to each task, compared with rankings from $F$ computed on that task. The benchmark found CIFAR-100 rankings uncorrelated with NinaPro and Darcy Flow. If our per-task analytic proxy recovers positive correlation with per-task trained accuracy where CIFAR-based rankings do not, that is the strongest evidence that patch statistics capture domain structure.
The third experiment is the failure analysis described above, together with a check on IIITD Newborn and Omniglot. There we search small residual topologies with SSF-CNN-style frozen structures and compare against SSF-CNN's reported 97.6% Omniglot 5-way 1-shot and 70.42% Newborn Rank-1 (SSF-CNN). This tests whether searching the topology adds anything beyond SSF-CNN's fixed backbone in the regime where unsupervised filters are already known to help.
References
- PCANet: A Simple Deep Learning Baseline for Image Classification?
- Convolutional Kernel Networks
- DARTS: Differentiable Architecture Search
- Large-Scale Evolution of Image Classifiers
- Evolving Search Space for Neural Architecture Search
- Agentic Neural Architecture Search
- Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
- Learning Structure and Strength of CNN Filters for Small Sample Size Training
- Emergence of Object-Selective Features in Unsupervised Feature Learning
- Unsupervised pathology image segmentation using representation learning with spherical k-means
- PCANet-II: When PCANet Meets the Second Order Pooling
- NAS-Bench-360: Benchmarking Diverse Tasks for Neural Architecture Search