Engineers who deploy vision on battery-powered sensors are in an awkward position. They can train cheaply on synthetic images, such as renderings or game-engine frames with free labels. They must run the model on a chip where every millijoule counts. And they usually have no real target images at design time. Current tools solve two of these constraints at a time. Hardware-aware architecture search finds networks that are fast and frugal on a specific chip, but it scores candidates only on clean, in-distribution data. Domain-robust architecture search finds networks that transfer across domains, but it ignores the chip, and some versions need real target data. This proposal asks whether a network's likely synthetic-to-real transfer can be read off the untrained network in a single pass, cheaply enough to sit inside a hardware-aware search next to a measured energy model.
The concrete bet is a training-free score computed on synthetic images only. It asks whether a freshly initialized network's internal features react more to changes in image structure or to changes in high-frequency texture. The two perturbations are defined in the Fourier domain, which splits an image into an amplitude part (roughly texture and contrast) and a phase part (roughly layout and edges). The motivation is empirical. Real sensors differ from simulators mainly in high-frequency amplitude, while phase carries semantics. A network whose wiring is already biased toward phase at initialization should be harder for simulator shortcuts to capture. We pair this score with a latency and energy predictor fitted to measurements from a real microcontroller. We then search for the energy-versus-transfer trade-off curve, report the energy spent on the search itself, and state in advance the result that would show the whole approach is unnecessary.
Core Claim
Within a hardware-feasible search space of small convolutional networks, architecture choice changes synthetic-to-real accuracy by an amount that strong augmentation does not erase. A zero-cost ratio of phase sensitivity to amplitude sensitivity, measured at initialization on synthetic images, ranks that transfer better than existing zero-cost proxies and better than noise-robustness proxies. Combining this ratio with measured energy yields networks that beat a hand-designed efficient backbone trained with the same augmentation at equal energy per inference, with search cost in CPU-hours rather than GPU-days.
The first sentence is an empirical premise, not an assumption. The program tests it first, because if it fails, the rest is moot.
Where This Sits
Three strands of prior work motivate the proposal.
PASTA shows that synthetic data under-represents high-frequency amplitude variance. It shows that perturbing amplitude in proportion to frequency, while leaving phase untouched, lifts DeepLabv3+ on GTA5→Cityscapes from 28.95 to 44.12 mIoU without any real data. PASTA fixes the architecture. We borrow its perturbation as a probe of architectures rather than as a training trick, and we also use it for training, so any architectural effect we find is measured on top of it.
G-NAS searches detection heads with a loss derived from neural tangent kernel theory that penalizes easy local features. It finds that larger kernels and skip connections generalize better to unseen weather. G-NAS reaches 33.5% average mAP on BDD100K single-domain generalization but never measures latency or energy.
HURRICANE shows why that omission matters. On a mobile DSP, large kernels are penalized. On a vision processing unit, squeeze-and-excitation blocks stall execution. Operators with equal FLOPs differ by more than 24x in latency. The robust wiring G-NAS prefers is exactly what edge hardware punishes. That tension is the reason to search both objectives jointly rather than in sequence.
Two strands supply method. HaeNAS pairs the Jacobian-covariance zero-cost proxy with a small learned predictor of on-device latency and energy, fitted to 1,000 profiled models, and reaches 80.7% ImageNet top-1 in about 160 GPU hours. We reuse its recipe of zero-cost accuracy scoring plus a profiled cost model, but replace its in-distribution scoring. EZNAS evolves zero-cost proxies from tensor statistics, reaching Kendall τ of 0.65 on NASBench-201 against 0.57 for Synflow and NASWOT, on one CPU in under a day. Its inputs are clean batches and Gaussian-perturbed batches, which is exactly the isotropic-noise view PASTA argues against.
Two strands constrain the design. EZNAS reports that zero-cost proxies fail to rank the top decile of architectures, so our success criterion must be about the top of the ranking, not overall correlation. NASDA searches with a source/target feature-discrepancy loss and reaches 96.7% on SYN SIGNS→GTSRB. It requires unlabeled target data during search plus about two days of adversarial training. We treat it as a target-data-using reference, not a like-for-like competitor. RNAS makes search robust to uniform random noise at 0.5 GPU days, but it is evaluated on adversarial attacks on CIFAR-10, not on domain shift. That is precisely the alternative our ablation must beat.
The Score
For a synthetic image $x$, let $\mathcal{A}(x)$ replace its Fourier amplitude with a PASTA-style perturbation that grows with frequency, and let $\mathcal{P}(x)$ apply a phase perturbation of matched pixel-space energy. Let $f_\theta$ be the network's penultimate features at random initialization $\theta$. The score of architecture $a$ is
$$ S(a) = \mathbb{E}_{x,\theta}\left[\log \frac{\lVert f_\theta(\mathcal{P}(x)) - f_\theta(x)\rVert}{\lVert f_\theta(\mathcal{A}(x)) - f_\theta(x)\rVert}\right], $$
where the expectation runs over a single mini-batch of synthetic images and a few random seeds. A high $S$ means the untrained network already responds more to structure than to texture.
Matching the pixel-space energy of the two perturbations matters. Without it, the score would simply reward networks that are insensitive to everything. That degenerate behaviour is one ablation below.
The search maximizes $S$, optionally combined with the Jacobian-covariance score so that the network remains trainable, subject to energy per inference predicted by a profiled model. We run an evolutionary search as in HaeNAS and keep the full energy-versus-score trade-off curve.
Research Program
The work proceeds in three stages, each of which can stop the project.
The first stage builds the ground truth. We define a search space of inverted-bottleneck blocks of the kind HaeNAS uses, varying kernel size, expansion ratio, depth, skip patterns and optional squeeze-and-excitation, restricted to operators the GreenWaves GAP9 neural engine executes. Accelerating Image-based Pest Detection on a Heterogeneous Multicore Microcontroller ran MobileNetV3-SSDLite on that chip at 4.85 mJ per inference; it chose the network by hand, used a 158-image real dataset and did no synthetic training. We sample about 200 architectures and train each fully on synthetic data with PASTA augmentation. We record synthetic validation accuracy, real-domain accuracy, and measured GAP9 latency and energy.
The benchmarks are SYN SIGNS→GTSRB and VisDA-C at reduced resolution. Both are cheap enough to train hundreds of models. The dense tasks, GTA5→Cityscapes and Sim10K→Cityscapes, are too expensive per model and are reserved for the final Pareto-front models. A subset of about 150 profiled architectures also fits the energy predictor.
The decisive premise test happens here. If, under PASTA training, synthetic-validation rank and real-domain rank agree closely across the space, then architecture does not add transfer risk beyond what augmentation handles. Concretely, the premise fails if Spearman correlation exceeds about 0.9 and the top-decile sets largely coincide. In that case the right recommendation is ordinary hardware-aware search on synthetic data plus PASTA, and we would publish that negative result.
The second stage scores all 200 architectures with $S$ and with the competing cheap signals. We measure Kendall τ against real-domain accuracy, and separately against the transfer gap (synthetic accuracy minus real accuracy). Because of the EZNAS caveat, we also report how many of the real top-decile architectures each signal places in its own top decile.
The third stage runs the full search on GAP9 and on one mobile-class platform, then evaluates the resulting trade-off curves. It reports total search energy, measured at the wall socket, against the per-inference savings multiplied by a deployment horizon. This answers the survey's point in A Survey of Methods for Low-Power Deep Learning and Computer Vision that search energy can exceed deployment savings.
Baselines
The naive reference point is MobileNetV3 compiled for GAP9, trained on synthetic data with PASTA and evaluated zero-shot on real data. This is what a practitioner would do today, and it is the reviewer's first alternative. It receives exactly the same training schedule, augmentation hyperparameters and quantization flow as every searched model. A searched model beats it only if it achieves higher real-domain accuracy at equal or lower measured energy per inference, or equal accuracy at clearly lower energy, across three training seeds. On the Codling Moth setting, the reference for the hand-designed network is 4.85 mJ per inference, from the pest-detection paper.
The strongest existing low-cost method is HaeNAS-style search: Jacobian-covariance scoring plus the same profiled energy predictor, run over the same space with the same evolutionary budget and population size. It differs from our method only in the accuracy signal. It is the most direct competitor because it already solves the hardware half at zero-cost prices. We beat it if our trade-off curve dominates its curve in real-domain accuracy at matched energy on at least two of the three classification and detection benchmarks. HaeNAS reports no sim-to-real results; no reported result was found on Lacuna for it on VisDA, GTSRB or Cityscapes, so its numbers here come from our own runs.
The strongest target-aware reference is NASDA, at 96.7% on SYN SIGNS→GTSRB and 76.8% on STL→CIFAR-10 in its paper. We cite rather than rerun its full adversarial pipeline, and we report how close a source-only, energy-constrained search gets to that figure. Closing most of the gap without target data would be a strong result. Falling short is expected and is not the claim.
For segmentation, the reference is PASTA's own DeepLabv3+ result of 44.12 mIoU on GTA5→Cityscapes. That model is far larger than anything that fits on GAP9. We therefore compare our Pareto models against a MobileNetV3-backbone segmenter trained with PASTA at matched energy, and cite 44.12 only as a ceiling.
Two ablations isolate the key ingredient. The first replaces $\mathcal{A}$ with isotropic Gaussian or uniform noise of matched energy, in the style of EZNAS's perturbed batches and RNAS-uniform. This tests whether the frequency structure matters. The second drops the phase numerator, which tests whether plain amplitude insensitivity, a possible stand-in for "insensitive to everything," is doing the work. We add a FLOPs-constrained version of our own search to measure how much the measured energy model contributes.
Why not just…?
Why not just train MobileNetV3 with PASTA and ship it? This is the real threat, and the premise test in stage one exists to settle it. PASTA may already remove most of the architecture-dependent transfer gap. HURRICANE's evidence that hand-designed mobile networks carry operators that stall specialized chips argues that a searched network should at least win on energy. Whether it also wins on transfer is unknown. If the rank-agreement check shows architecture barely matters for transfer, we will say so. The contribution then shrinks to hardware-aware search on synthetic data, which HaeNAS already covers.
Why not just run standard hardware-aware search on synthetic data, then fine-tune on a little real data? Many edge deployments, such as remote crop sensors, have no labeled real data before deployment, which removes the fine-tuning step. Where some real data does exist, fine-tuning is legitimate. We will include a small-real-data fine-tuning arm for the HaeNAS model on GTSRB, to show how much labeled real data is needed to match our source-only result. G-NAS's argument that topology fixes a bias fine-tuning cannot undo is a hypothesis, not a demonstration, and this arm tests it.
Why not just select architectures by short training plus a perturbed synthetic validation set? A few epochs of training, scored on PASTA-perturbed synthetic validation images, is a sensible middle-cost estimator in the spirit of budgeted performance estimation. It might rank transfer better than any initialization-time score. We run it at one, three and ten epochs as an additional signal in stage two. If it clearly beats $S$ on top-decile recall, the honest recommendation is to use it, with $S$ kept as a cheap pre-filter. The cost comparison, CPU-minutes against GPU-hours per candidate, will be reported alongside.
Why not just constrain FLOPs instead of measuring energy? HURRICANE's 24x latency spread among equal-FLOP operators, and the survey's observation that memory traffic dominates energy, make FLOPs a poor constraint on heterogeneous chips like GAP9. The FLOPs-constrained ablation will quantify how far off it is on our specific search space.
Decisive Checks
Three outcomes decide the project.
First, the premise check: real-domain ranks must diverge meaningfully from synthetic-validation ranks under PASTA training, otherwise the proxy is unnecessary.
Second, the proxy check: $S$ must beat the Jacobian-covariance, Synflow and matched Gaussian-noise proxies on top-decile recall of real-domain accuracy, not only on overall Kendall τ, on both SYN SIGNS→GTSRB and VisDA-C.
Third, the deployment check: on GAP9, the searched models must dominate MobileNetV3 with PASTA and the HaeNAS-style search on the measured energy-versus-real-accuracy curve. The advantage must carry over to Sim10K→Cityscapes detection, and the total search energy must be small enough that the per-inference savings repay it within a stated deployment horizon.
A pass on all three would give practitioners a synthetic-only, chip-specific way to choose networks. Failing the first check is itself a useful finding, because it would show that augmentation, not architecture, is where sim-to-real effort belongs.
References
- Pasta: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization
- G-NAS: Generalizable Neural Architecture Search for Single Domain Generalization Object Detection
- Fast Hardware-Aware Neural Architecture Search
- HaeNAS: Hardware-Aware Efficient Neural Architecture Search via Zero-Cost Proxy
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Network Architecture Search for Domain Adaptation
- Robust Neural Architecture Search
- Accelerating Image-based Pest Detection on a Heterogeneous Multicore Microcontroller
- A Survey of Methods for Low-Power Deep Learning and Computer Vision
- Rethinking Performance Estimation in Neural Architecture Search