Predicting Which Architectures Survive Integer-Only Training Before Training Them

Some chips have no floating-point hardware: microcontrollers, small FPGAs, many edge accelerators. To train a network on such a chip, every step has to run in integers, including the backward pass and the weight update. Methods for this now exist. NITI trains small convolutional networks entirely in integers by giving each layer a shared power-of-two scale. NITRO-D removes floating point from deep CNNs by splitting them into blocks that each learn from a local loss. Both, however, train hand-picked networks only. When training from scratch they still lose accuracy relative to 32-bit floating point, from 0.15% to 7% on CIFAR-10 for NITRO-D. The usual suspect is update underflow: once the step $\eta g$ falls below the smallest change an integer weight can represent, rounding erases it and the weight freezes. This is the "stuck state" catalogued in the underflow literature. Whether that freezing depends on the architecture, and so whether choosing the network well could close the gap, is unknown.

This note proposes to settle that question cheaply and then build on the answer. We will first measure how differently architectures rank when trained in integers versus floating point, using a subset of NAS-Bench-201 retrained under integer-only arithmetic. We will then test a zero-cost proxy for integer trainability. A zero-cost proxy is a score computed from one minibatch at initialization, with no training. Ours runs a single integer forward and backward pass. It then measures what fraction of each layer's weight updates would survive rounding as the gradients shrink. The claim is that this "update survival" score predicts final integer-training accuracy better than existing proxies. That includes existing proxies naively recomputed in integer arithmetic. The study is also built to fail cleanly: if floating-point rankings already predict integer rankings, we will say so.

Core Claim

Existing zero-cost proxies such as synflow, snip, jacob_cov, nwot and the evolved EZNAS-A score were designed and validated against floating-point training outcomes (Zero-Cost Proxies for Lightweight NAS; EZNAS; NAS-Bench-Suite-Zero). They measure how strongly gradients flow and how expressive the network is. None of them asks whether those gradients, once quantized and scaled by a learning rate, are large enough to move integer weights at all.

We hypothesize that, under integer-only training, an architecture's final accuracy depends heavily on a quantity floating-point proxies cannot see: the margin between its per-layer update magnitudes and the weight quantization step. We further hypothesize that this margin can be estimated at step zero by sweeping a gradient-shrinkage factor. Let $\Delta_l$ be layer $l$'s weight quantization step, set by its block exponent. Let $\eta$ be the learning rate, and let $g$ be a weight's gradient from one integer backward pass. We define the survival curve as

$$S_l(c) = \frac{1}{|W_l|}\sum_{w \in W_l} \mathbf{1}\left[\, c\,\eta\,|g_w| \ge \Delta_l/2 \,\right],$$

which is the fraction of layer $l$'s weights $W_l$ whose update would not round to zero if gradients shrank by a factor $c \in (0,1]$. The architecture's score is the area under $\min_l S_l(c)$ over $\log c$. Taking the minimum over layers captures that a single frozen layer can bottleneck training. Integrating over $c$ stands in for the gradient decay of late training, where stagnation is reported to bite (Chmiel et al.). Because survival alone can reward trivially small or skip-heavy networks, the final proxy pairs it with one expressivity term, nwot. NAS-Bench-Suite-Zero found nwot among the most robust single proxies.

The difference from prior intersection work is specific. OQAT and ∇QDARTS search for architectures that are accurate at inference under quantization. Both run simulated quantization inside floating-point training on GPUs, and both need expensive training of a shared network that contains every candidate (a supernet). We target trainability under native integer arithmetic, and we spend no training on the search itself.

Research Program

The first deliverable is a small integer-trained table. We will sample 1,000 of the 15,625 NAS-Bench-201 cells, stratified by floating-point accuracy, with oversampling of the top decile. Each will be trained from scratch on CIFAR-10 under two integer regimes:

NAS-Bench-201 cells contain batch normalization, which needs floating-point division. We will replace it with NITRO-D's integer rescaling (NITRO-D) and also retrain those same 1,000 cells in floating point with that replacement. This separates the effect of the normalization change from the effect of integer arithmetic. Each integer run will be done with both deterministic rounding and NITI's pseudo-stochastic rounding. Stochastic rounding removes systematic underflow in expectation, so it gives a built-in control: under stochastic rounding, the survival proxy's advantage should shrink.

A 300-cell subset will be repeated on CIFAR-100 and ImageNet-16-120. This tests whether the findings transfer across tasks.

The second deliverable is the analysis on that table. It has three parts:

  1. The rank correlation between floating-point and integer accuracy, overall and within the top 10%.
  2. Logged per-layer frozen-weight fractions during training. These test whether stagnation actually explains the accuracy losses. If it does not, the proxy has no mechanism behind it.
  3. A comparison of the survival proxy against all baselines below. Rank correlation is the measure of prediction quality, and simulated search on the table is the measure of practical value.

Baselines

Floating-point lookup. The naive reference is to do nothing new: rank candidates by their published NAS-Bench-201 floating-point accuracy, which costs zero compute. If its Spearman correlation with integer accuracy is high, every other method must beat that number to matter.

Existing floating-point zero-cost proxies. The strongest existing training-free competitor is the voting ensemble of synflow, jacob_cov and snip. It reported a Spearman correlation of 0.82 on NAS-Bench-201 against floating-point targets (Zero-Cost Proxies for Lightweight NAS). We will also run EZNAS-A, which reported a Kendall tau of 0.65 on NAS-Bench-201 CIFAR-10, against 0.57 for NASWOT and synflow (EZNAS). No reported result was found on Lacuna for any proxy against integer-trained targets; those correlations are what this work produces.

Fairness conditions. All proxies will use the same minibatch size, the same seeds, and the same 1,000 cells. The ensemble members will be re-tuned on a held-out 200-cell split with the same search budget given to our proxy's single free choice, the range of $c$.

Integer-arithmetic versions of existing proxies. We will recompute every existing proxy using the integer forward and backward pass. This is the simplest way to "make proxies precision-aware." It is also the ablation that isolates our key ingredient: if integer grad_norm matches survival, then the update-versus-step comparison adds nothing.

Ablations of our own proxy. We will run survival at $c=1$ only, without the sweep, and survival without nwot. These test whether each component is necessary.

What counts as winning. Beating the baselines means a higher Kendall tau than the best of them on integer accuracy, with bootstrap confidence intervals over cells that do not overlap. It also means lower search regret, the accuracy gap between the chosen architecture and the true best in the table, within the top decile.

Why not just…?

Why not just search in floating point and train the winner in integers? This is a real threat. OQAT's ranking gap was measured for 2-bit inference: a 2-bit MobileNetV3 with LSQ reached 52.8% ImageNet top-1, against 61.6% for a quantization-co-searched model (OQAT). It need not carry over to 8- or 16-bit integer training. Fixed-Point Back-Propagation Training reached a 0.02% average loss on standard architectures with adaptive bit-widths. That suggests the gap may be small when precision is allowed to adapt. Our first analysis measures this rank correlation directly. If top-decile correlation exceeds roughly 0.9, the proxy line is moot, and we report the table and that negative finding instead.

Why not just train each candidate for a few integer epochs? Reduced training is the natural strong competitor. It ranked worse than zero-cost proxies in floating point, with Spearman 0.61 for EcoNAS versus 0.82 for the ensemble, and its wall-clock savings fall well short of its FLOP savings (Zero-Cost Proxies for Lightweight NAS). In the integer setting, however, early epochs might expose stagnation better than a step-zero score. We include 1-, 2- and 5-epoch integer training as baselines, matched in wall-clock time to the proxy where possible. We will report the cost at which reduced training overtakes the proxy.

Why not just use parameter count or FLOPs? NAS-Bench-Suite-Zero found flops to be a robust single proxy. OQAT, however, found that deeper low-precision networks accumulate error, which would invert a size bias. Both are run as baselines. If size alone matches survival, then underflow is not architecture-dependent in a way size fails to capture, and we would report that.

Why not just fix underflow with stochastic rounding and skip architecture selection? If pseudo-stochastic rounding closes the integer-versus-floating-point gap uniformly across cells, architecture choice matters little for integer training. Running every cell under both rounding schemes answers this directly.

Decisive Checks

The project stands or falls on four observations from the 1,000-cell table. First, floating-point-to-integer rank correlation must be clearly below 1 within the top decile under deterministic rounding. Second, the frozen-weight fraction logged during training must correlate with each cell's accuracy loss relative to its floating-point counterpart. Third, the survival proxy must outperform integer-recomputed grad_norm and the ensemble in Kendall tau and in top-decile regret. Fourth, its advantage must shrink under stochastic rounding.

If all four hold on CIFAR-10 and the proxy's ranking transfers to CIFAR-100 and ImageNet-16-120, the result is the first evidence that architecture choice can partly substitute for extra bits in integer-only training. It would also provide a search signal cheap enough to run on the device itself.