Hardware-aware architecture search tries to find neural networks that are both accurate and fast on a specific chip. Scoring accuracy has become cheap: training-free "zero-cost proxies" rank candidate networks from a single batch in milliseconds. Scoring speed has not become cheap. Theoretical operation counts (FLOPs) track measured latency poorly, so current methods profile hundreds to thousands of real networks on each target device and fit a lookup table or a learned predictor to the results. That per-device profiling now costs more than accuracy estimation. We propose replacing it with a small analytical model of how a CPU's SIMD units execute each layer. SIMD, or single-instruction-multiple-data, refers to vector instructions that process 4, 8, or 16 numbers at once. The model is calibrated by a few minutes of single-layer microbenchmarks rather than by profiling whole networks.
The core intuition is simple. A vector unit processes channels in fixed-size groups called "lanes." A layer with 40 channels on a machine with 16 float lanes pays for 48 channels, because the last group is padded with useless data. On a machine with 4 lanes, the same layer pays for exactly 40. FLOP counts cannot represent this rounding, which depends on both the layer and the device. Channel-width search spaces are full of widths that fall on one side of a lane boundary or the other. Our claim is that most of the device-specific latency structure in such spaces comes from this rounding together with a memory-bandwidth ceiling and a per-layer overhead term. If so, a profiling-free model built from lane counts can rank architectures nearly as well as predictors trained on many measured networks.
Core Claim
For convolutional search spaces on CPUs with 128- to 512-bit vector units, a latency model with three ingredients can match empirical predictors on rank correlation with measured latency and on the quality of the speed–accuracy trade-off curves that search returns. The first ingredient is lane-padded work: each layer's multiply-accumulates are rounded up along whichever axis the backend actually vectorizes. The second is a roofline bound, meaning the layer runs no faster than either its arithmetic rate or its memory-traffic rate allows. The third is a constant per-layer launch cost.
The model needs no full-network profiling. Instead, each device answers about a hundred single-layer timing queries, and their cost is measured in device-minutes rather than device-hours. Per layer $\ell$ we write
$$ t_\ell = \max\!\left(\frac{M_\ell \cdot \lceil C_\ell / L \rceil L / C_\ell}{P},\ \frac{B_\ell}{W}\right) + o, $$
where $M_\ell$ is the layer's multiply-accumulate count and $C_\ell$ is its size along the vectorized axis. $L$ is the effective lane period, $P$ is peak vector throughput, $B_\ell$ is bytes moved, $W$ is sustained memory bandwidth, and $o$ is per-layer overhead. Predicted network latency is $\sum_\ell t_\ell$.
The device-specific constants $L$, $P$, $W$ and $o$ are read off a width sweep. We time a single convolution while varying its channel count from 1 to 64, and the period of the resulting sawtooth reveals $L$ for that device and backend. This matters because $L$ depends on the backend's data layout as well as the register width. Dolz et al. find that the channels-first (NCHW) layout is generally more efficient than channels-last (NHWC) for Winograd transforms because it gives better memory contiguity, so different backends can vectorize different axes.
Where This Sits
Three papers motivate the gap. HW-NAS-Bench measured latency for tens of thousands of architectures across six devices. It found that FLOPs-to-latency rank correlation falls below 0.5 on several devices and to 0.18 on the Edge TPU, but it treats latency as a black-box measurement. HaeNAS pairs a Jacobian-covariance zero-cost proxy with an MLP latency predictor trained on 1,000 profiled architectures. It reaches 80.7% ImageNet top-1, but only with that profiling budget. One Proxy Device Is Enough cuts profiling to 30 to 80 target samples by linearly adapting a predictor from one device to another. Our work removes the full-network samples entirely and explains why rankings shift between devices. AdaProxy cannot do either, because it never models the hardware.
Two lines of work supply the method. The Winograd kernel papers, AirWino and Dolz et al., show that throughput is governed mechanically by arithmetic intensity, register blocking and layout contiguity. Dolz et al. also show that processing padded "garbage" lanes beats scalar boundary handling, which is exactly the rounding our model counts. AirWino goes further and already uses an analytical performance model to choose kernel modes. Both papers apply this reasoning to fixed human-designed networks. We apply it as a fitness signal inside search. ShuffleNet V2 supplies the memory-access-cost argument behind our bandwidth term.
Two findings constrain the design. Zero-Cost Proxies for Lightweight NAS reports that cutting proxy FLOPs by 64× gave only a 4× wall-clock gain, because launch and synchronization overhead dominate small workloads. This is why the overhead term $o$ is mandatory. It also warns that lane effects could be washed out end to end, and we test that directly.
Research Program
We will first build the calibration harness. It runs single-layer width sweeps in each device's deployment backend: PyTorch with oneDNN on x86 and with the ARM Compute Library on ARM. From each sweep it fits $L$, $P$, $W$ and $o$ by least squares, with $L$ chosen from candidate periods by best fit.
Second, we will build ground truth by measuring end-to-end latency on four CPUs spanning lane widths. These are a Raspberry Pi 5 and an AWS Graviton2, both with 128-bit NEON; an Intel Skylake Xeon with AVX-512; and, if access allows, a Fujitsu A64FX with 512-bit SVE. On each we measure all 32,768 architectures of the NATS-Bench size space, which draws widths from {8, …, 64}. In 32-bit floats those widths are always aligned for 4-lane NEON but often misaligned for 16-lane AVX-512. That gives a built-in prediction: the lane term should matter on Skylake and A64FX and be nearly inert on the ARM boards. We measure at batch sizes 1 and 32, because overhead and bandwidth balance differently in the two regimes.
Third, we plug each latency model into Regularized Evolution. NATS-Bench found this search method consistently matches or beats reinforcement-learning and weight-sharing methods. It runs with the SWAP proxy from SWAP-NAS for accuracy. We compare the speed–accuracy fronts it finds against the true front computed from tabular accuracy and measured latency.
Baselines
The naive reference is FLOPs, or equivalently FLOPs plus parameter count, used as the latency signal at zero profiling cost. It is the default in much of the search literature. On HW-NAS-Bench its reported Kendall correlation with latency was below 0.5 on several devices and as low as 0.18 on the Edge TPU. To beat it, our model must exceed it on every CPU at the same zero full-network profiling budget.
The ablation that isolates our key ingredient is the same roofline-plus-overhead model with the ceiling term removed, so that $L=1$. It receives identical microbenchmark calibration. If our model does not clearly beat it on the AVX-512 and SVE machines in the size space, the lane claim is wrong. We also expect the two to tie on the NEON boards, as explained above; failing that would also undercut the mechanism.
The strongest existing competitor is AdaProxy from One Proxy Device Is Enough. It adapts a base predictor to a new device with 30 to 80 measured networks, and the paper reports cross-platform rank correlation rising from 0.6–0.8 to above 0.9. We give it a fair budget of measured target networks, matched to our microbenchmark budget in device-seconds and also at its native 80 samples. Its base predictor is trained on one of our four CPUs, and we evaluate on the other three.
A second strong competitor is an MLP predictor trained on 1,000 profiled architectures, as in HaeNAS. It shows what heavy profiling buys. We train it on the same feature encoding used in HaeNAS, and it gets the same hyperparameter-tuning budget as our model's calibration fit.
There is no reported result found on Lacuna for latency prediction on the NATS-Bench size space on CPUs, so all baseline numbers on that space are our own. Beating AdaProxy means matching its rank correlation within 0.02 using no full-network measurements, or exceeding it at equal device time. We will not claim to beat the 1,000-sample MLP outright. The target is to come within 0.05 Kendall of it and to produce speed–accuracy fronts that are statistically indistinguishable from it.
Why not just…?
Why not just restrict the search space to lane-multiple widths? For a single known device this is a cheap and real alternative, and we test it. We compute the true speed–accuracy front of the NATS size space restricted to multiples of 16 against the unrestricted front. On NEON the restriction discards aligned widths for nothing. On AVX-512 it may discard accurate mid-size networks whose padding cost is small. Restriction also fixes the space to one device, whereas the model scores any width on any device. If the restricted front matches the unrestricted front on all devices, we will report that the simple rule suffices for this space.
Why not just use AdaProxy? On similar device pairs it is likely already very good; its authors report above 0.98 Spearman within smartphone CPUs. Its weakness is that it needs the physical target and a base device that is similar enough. HW-NAS-Bench shows that models searched for one platform can be about 50% slower on another. Our cross-family transfer experiment between NEON and AVX-512 machines settles whether the analytical model or the linear transfer generalizes better. If AdaProxy wins at equal device time, the contribution narrows to explaining why its transfers sometimes fail.
Why not just use a roofline model, or FLOPs plus memory traffic? This is exactly our ablation without the lane term, and it may capture most of the variance in topology spaces where widths are fixed. Our prediction is that it fails specifically where widths cross lane boundaries. The size-space versus topology-space contrast and the NEON versus AVX-512 contrast test this directly.
Why not just vectorize the proxies themselves to cut search cost? SWAP-NAS already reports about 6 minutes of search on CIFAR-10. Further speeding up accuracy scoring saves little, while device profiling remains a cost of hours and a dependence on hardware access.
Decisive Checks
The first check comes before anything else: whether lane padding survives to end-to-end latency. Using the Skylake measurements, we regress network latency on lane-padded versus unpadded work while controlling for layer count. If the sawtooth seen in single layers is buried by framework overhead at both batch sizes, the central claim fails, and we will report it as a negative result consistent with the overhead finding in Zero-Cost Proxies for Lightweight NAS.
The second check is rank correlation with measured latency, as Kendall and Spearman, for every model on every device. We run it on the NATS size space and also on NAS-Bench-201 using HW-NAS-Bench's published Raspberry Pi 4 measurements, which serve as external ground truth we did not collect.
The third check is search outcome. We measure the hypervolume of the front found by Regularized Evolution under each latency model, relative to the true front. The comparison is plotted against total device-seconds spent on profiling, because profiling cost is the quantity the proposal aims to reduce.
References
- Efficient and portable Winograd convolutions for multi-core processors
- HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
- HaeNAS: Hardware-Aware Efficient Neural Architecture Search via Zero-Cost Proxy
- One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search
- AirWino: Optimized Winograd Convolution for Accelerating CNN Inference on ARMv8 Processors
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
- Zero-Cost Proxies for Lightweight NAS
- NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
- SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS