Searching for Architectures That Stay Correct Under Bit-Flips

Neural networks running on small accelerators sometimes suffer a random bit-flip in a register or memory cell, caused by radiation, voltage noise, or heat. The chip does not crash. The network simply returns a confident wrong answer, which the field calls silent data corruption. Two cheap fixes exist today. One is to clip each layer's activations to the range seen on clean data, so a flipped exponent bit cannot blow a value up to $10^{38}$. The other is to search automatically for network shapes that are fast on a given chip, a process called hardware-aware neural architecture search, and to make that search affordable by scoring untrained candidates with "zero-cost proxies," which are one-batch statistics computed at initialization. These two fixes have never been combined. Hardware-aware search optimizes latency and memory and ignores reliability. Reliability work hardens fixed, human-designed networks.

This note asks a narrow question that decides whether combining them is worthwhile at all: once activation clipping has been applied, does the choice of architecture still change the residual corruption rate by a meaningful factor at equal latency and accuracy? If it does, we propose a zero-cost proxy that predicts this residual risk from an untrained network. That proxy would let a latency-constrained search pick reliable architectures for a few GPU hours instead of thousands. Model extraction, where an attacker copies a deployed model by querying it, is not a co-equal goal here. It enters as an evaluation axis: we check whether architectures chosen for reliability are easier or harder to steal, so the search does not trade one deployment risk for another without anyone noticing.

Core Claim

We claim that per-inference corruption risk factors into two parts. The first is exposure, meaning how many storage and compute sites a fault can land in, which FLOPs and memory footprint already roughly track. The second is propagation, meaning how far a bounded fault at one site travels to the output once clipping is in place. Architecture topology shapes propagation in ways FLOPs cannot see: skip connections, pooling, fan-out, and depth all matter. We further claim that propagation can be estimated at initialization with a few forward passes. Concretely, for architecture $a$ we define

$$ P(a) = \mathbb{E}_{x,\,l,\,i}\left[ \frac{\lVert f_a(x;\ \text{site } (l,i) \leftarrow b_l) - f_a(x) \rVert}{\lVert f_a(x) \rVert} \right], \qquad R(a) = N(a)\cdot P(a). $$

The symbols are as follows. $f_a$ is the clipped network evaluated after the scale-insensitive BatchNorm recalibration that Zen-Score uses. Site $(l,i)$ is a randomly chosen activation $i$ in layer $l$, set to its clip bound $b_l$, which is the worst value a clipped fault can produce. $N(a)$ is the number of fault-exposed sites per inference. $R(a)$ is the proxy for residual corruption risk. The product form follows from treating faults as independent per site. Under that assumption, the expected number of corrupting faults is the number of sites times the per-site corruption probability.

This proxy is new in three respects. Existing zero-cost proxies, which NAS-Bench-Suite-Zero benchmarks across 13 scores and 28 tasks, all target clean accuracy. Existing reliability measurements require training plus fault-injection campaigns: over 2 million register-level flips in the Ethos-U55 study, and runtime profiling in Ranger. Finally, the proxy targets the risk that remains after clipping, not before.

Relation to Prior Work

The reliability literature motivates the gap. Ranger inserts clip bounds profiled on clean data and cuts classifier corruption from 14.92% to 0.44% at 0.53% runtime overhead. It does this for six fixed networks and reports that the reduction varies by model, and that it is weaker for the Dave steering model because of its atan output. That variation is our first hint that architecture matters even after clipping. Global Clipper and the range-monitoring work of Geissler et al. show the same kind of bounding works inside attention blocks and in CNNs. Meanwhile, the Winograd fault-tolerance studies show that how an operator is computed changes how errors spread. That is direct evidence that structure, not just size, governs propagation.

The Ethos-U55 characterization constrains our claims. It finds that the area of a hardware block does not predict its vulnerability, and that a speech network (Wav2Letter) was far more fragile than vision networks on identical silicon. We take this as a warning that a software-level proxy can only rank architectures relative to one another. It cannot certify an absolute safety level. We therefore never claim to meet a safety-standard threshold, only to rank candidates.

The search literature supplies the method. HW-NAS-Bench supplies measured latency and energy for all 15,625 NAS-Bench-201 cells on six devices, and shows that FLOPs can rank-correlate with Edge TPU latency at Kendall τ as low as 0.18. That is the same failure mode we expect FLOPs to show for reliability. Zen-NAS supplies the BatchNorm recalibration that keeps our forward passes at initialization numerically stable. DUCCIO supplies the hinge-style constraint $\max(0, R_j - T_j)$ that lets us impose a risk ceiling without tuning a trade-off weight. In DUCCIO this kind of tuning previously cost about ten times more search time. None of these three papers measures faults.

The extraction literature sets the evaluation lens. Breaking TinyML copies INT8 edge models with hard-label queries, reaching 82.7–83.8% agreement on CIFAR-10 within 50,000 queries. We reuse that protocol to test whether reliability-chosen architectures are more copyable, for example because clipping produces smoother functions. CMIM is relevant because it deliberately collapses within-class outputs, which could sharpen decision margins in the way bit-flips exploit. We include it as a defended variant rather than as a core component. NASPY, which recovers architectures through cache timing of matrix-multiply calls, cannot see non-matrix operations such as clipping. We note this only as a side observation and do not build on it.

Research Program

The work has three stages, and the first is a gate.

The first stage builds ground truth. We train a stratified sample of about 500 NAS-Bench-201 cells on CIFAR-10 and CIFAR-100, stratified by FLOPs and by HW-NAS-Bench latency on Edge TPU, Pixel 3, and Jetson TX2. We then inject single-bit flips into weights and activations in software under three number formats: FP32, 16-bit fixed point, and INT8, the last being the standard in MLPerf Tiny. We record corruption rate per inference, defined as a top-1 change, both without clipping and with Ranger-style clipping. This produces the first reliability table aligned with a hardware-aware NAS benchmark.

The second stage computes $P(a)$ and $R(a)$ at initialization for all 15,625 cells. We measure the rank correlation with the ground-truth subset and compare it against every competing predictor described below. If the proxy at initialization fails, we test a one-epoch variant before abandoning the approach. We report its cost honestly, because it is no longer zero-cost.

The third stage runs a DUCCIO-style constrained search. The objective combines a clean-accuracy score with hinge penalties for a latency ceiling on each device and for a ceiling on $R(a)$. We then fully train and fault-inject the selected architectures. Finally, we transfer the procedure to the MLPerf Tiny Visual Wake Words task at INT8 to see whether the ranking survives quantization and a different search space.

Baselines

The naive reference point is the obvious practice today: run a latency-constrained search that ignores reliability, then apply Ranger to whatever it returns. We run it with the same accuracy proxy, the same HW-NAS-Bench latency ceilings, and the same search budget as our method. Its measured post-clipping corruption rate is the number to beat at matched latency and clean accuracy. For scale, Ranger reports a 0.44% average corruption rate on six classifiers (Ranger). However, no reported result was found on Lacuna for corruption rates of NAS-Bench-201 architectures, so this baseline must be measured inside our own campaign. Beating it means a statistically reliable reduction in post-clipping corruption, which we will treat as at least 2× at the same latency tier, with clean accuracy within 0.5 points.

The strongest existing predictor is the learned ensemble of NAS-Bench-Suite-Zero. It fits XGBoost on all 13 zero-cost proxies and improves Spearman correlation with clean accuracy by up to 42% over single proxies. We retrain that same ensemble to predict post-clipping corruption rate on the same training split of our ground-truth table. This gives it the same data and the same tuning budget as any regressor we fit on top of $R(a)$. Our proxy beats it if its held-out Kendall τ is higher. We also accept a weaker result: adding $R(a)$ as a feature to the ensemble raises τ significantly, which would show that it carries information the 13 existing proxies lack.

Two ablations isolate the key ingredient. The exposure-only predictor uses $N(a)$ alone, or FLOPs, or parameter count. If it matches $R(a)$, the propagation term is worthless and the problem reduces to "pick smaller networks." The propagation-only predictor uses $P(a)$ alone. It tells us whether per-site propagation separates architectures at matched size. A third ablation removes the clipping inside $P(a)$. It tests whether predicting residual risk differs from predicting raw risk, and that distinction is the central claim.

For the extraction axis, we copy each final architecture under the hard-label Copycat protocol of Breaking TinyML, with a 50,000-query budget. Their reported agreement of 82.7–83.8% on CIFAR-10 is the reference. A reliability-chosen architecture "passes" if its copy agreement does not rise above that of the latency-only baseline by more than run-to-run noise.

Why not just…?

Why not just apply Ranger to whatever architecture a latency search returns? This is the real threat. If clipping flattens differences between architectures, then reliability-aware search is pointless. Ranger's own per-model variation and its weaker results on the steering model suggest architecture still matters, but six models are not evidence across a search space. The first stage answers the question directly. If post-clipping corruption varies by less than 2× across architectures within each latency tier, we stop. We then report that negative result, which itself tells practitioners that clipping is sufficient.

Why not just use FLOPs or the number of exposed sites as the reliability score? Exposure probably explains much of the variance, since more sites means more chances to be hit. If it explains nearly all of it, our proxy adds nothing. The exposure-only ablation tests this head-on. The HW-NAS-Bench result, where FLOPs track Edge TPU latency at τ as low as 0.18, shows that size-based proxies can fail badly when topology drives the outcome. That is why we expect, but do not assume, that propagation matters.

Why not just fault-aware training or hardware redundancy? Both act at a different layer of the problem and can be combined with our approach rather than replacing it. Fault-aware retraining works, but Reduce exists precisely because its GPU cost is severe, and it applies per trained model rather than during search. Full dual-core lock-step costs 100% extra silicon area, against about 38% for selective hardening (Ethos-U55), which rules it out on microcontrollers. Starting from an architecture with lower residual risk should lower the work left for either method. We test this on the final architectures by measuring how many fault-aware fine-tuning epochs each needs to reach a fixed corruption target.

Why not just train candidates and inject faults directly? Once a model is trained, software fault injection is cheap. The expensive part is training each candidate. Doing this for every search candidate would bring back exactly the thousands of GPU hours that zero-cost search was built to avoid. We do pay this cost once, for 500 cells, to build ground truth. The proxy then has to justify itself by ranking the other 15,000-plus cells for minutes of compute.

Decisive Checks

The project succeeds or fails on four measurements. First, within each latency tier on each device, the post-clipping corruption rate must vary across architectures by at least 2×. If it does not, we publish the negative result and stop. Second, $R(a)$ computed at initialization must beat both the FLOPs and $N(a)$ ablations and the retrained 13-proxy ensemble in held-out Kendall τ, or measurably improve that ensemble when added as a feature. Third, the constrained search, run for no more than a few GPU hours, must return architectures whose measured post-clipping corruption is at least 2× lower than the latency-only-plus-Ranger baseline at matched latency and within 0.5 points of its clean accuracy. This must hold in FP32 and also at INT8 on Visual Wake Words. Fourth, those architectures must not be measurably easier to copy under the 50,000-query hard-label attack. A CMIM-trained variant will be checked for the opposite risk: that suppressing extraction raises fault vulnerability.