Deployed image networks often run slowly for reasons that operation counts miss. A chain of small convolutions separated by activation functions (simple nonlinear functions such as ReLU, applied element by element) forces the hardware to write every intermediate result back to memory and launch a separate kernel for each step. If an activation is removed, the linear layers on either side can be multiplied together into one dense layer. Batch normalization layers that were trapped between activations also become foldable into that layer. The result is often much faster, even when it has more arithmetic. The cost is accuracy, and that cost is unpredictable. The same channel-bypass trick that speeds up ViT-Large without loss drops DeiT-Tiny from 72.1% to 64.2% (RePaViT). Today the only way to find out which activations a network can spare is to train: differentiable mask search followed by fine-tuning (DepthShrinker), or a 120-epoch schedule that fades out nonlinearities (Double-Win NAS).
This proposal splits the question into two halves. The latency half needs no prediction at all. The collapsed deployment graph can be built exactly from randomly initialized weights and profiled on the target device, because runtime does not depend on weight values. The accuracy half is where a training-free estimate is actually needed, and there existing zero-cost proxies are likely to be wrong in a specific, testable direction. A zero-cost proxy is a score computed from one or two minibatches at random initialization that is meant to rank architectures by their eventual trained accuracy. We propose to build a small benchmark of trained "linearized" variants, show where current proxies fail on it, and test a forward-pass expressivity-gap score designed to predict the accuracy cost of removing a specific set of activations.
Core Claim
Existing proxies are poorly suited to linearization because of two structural facts.
The first is a function-equivalence fact. Structural reparameterization (rewriting a trained multi-branch block as one equivalent layer) and batch-normalization folding are exact rewrites of the same function (RepVGG, To Fold or Not to Fold). Any forward-only expressivity score therefore gives identical values on the training graph and the collapsed graph. The worry that proxies are "computed on a graph that won't exist at inference" matters only for gradient-based proxies, and for those the training graph is the correct one to score, since it is the graph that gets optimized.
The second is a gradient-noise fact. Removing a ReLU makes a block closer to linear, which tends to make per-sample gradients more consistent. Gradient signal-to-noise proxies such as ZiCo may therefore reward linearization even as expressive capacity falls.
Our hypothesis follows from these two facts. The accuracy cost of removing an activation set $S$ is best predicted by the drop in forward expressivity that $S$ causes, measured at initialization on the uncollapsed network. It is not well predicted by any absolute proxy score of the linearized network. We define
$$\Delta_E(S) = \mathbb{E}_{\theta_0}\big[E(f_{\theta_0}) - E(f_{\theta_0}^{\setminus S})\big],$$
where $f_{\theta_0}$ is the network at random initialization $\theta_0$, $f^{\setminus S}_{\theta_0}$ is the same weights with the activations in $S$ replaced by identity, $E$ is an expressivity measure (we test log effective rank of penultimate features over a minibatch, PCA isotropy as in AZ-NAS, and linear-region counts as in Understanding and Accelerating NAS With Training-Free Metrics), and the expectation runs over a few seeds. Because the difference is taken under shared weights, it isolates the contribution of the removed nonlinearities. It also cancels much of the initialization sensitivity that ZiCo reports.
The selection objective then trades measured latency saved, $\Delta L(S)$ (profiled on the collapsed, batch-norm-folded graph), against predicted accuracy loss. We choose $S$ to maximize $\Delta L(S) - \lambda\,\Delta_E(S)$ under a latency target, with the weight $\lambda$ set once on held-out architectures.
Research Program
The first step is a collapse compiler that produces the deployment graph for any removal set. Consecutive linear operations left without activations between them are multiplied into one dense layer, following the DepthShrinker and Double-Win NAS construction. Batch normalization is then folded using the necessary-and-sufficient graph condition of To Fold or Not to Fold. That paper supplies the method for folding and also motivates the latency side. Its BaN-OFF algorithm explicitly cannot fold a batch normalization trapped between two ReLUs, so part of the latency gain from removing an activation is that it frees previously unfoldable normalizations. Because we measure latency rather than model it, this effect is captured automatically. The effect is device-dependent: HW-NAS-Bench reports FLOPs–latency rank correlation as low as 0.18 on Edge TPU. We therefore profile each collapsed graph on a desktop GPU and on at least one edge device (Raspberry Pi 4 or a Jetson board).
The second step is ground truth. Existing benchmarks report nothing about linearized variants, so we must train some. We use two search spaces. The first is an inverted-residual (MobileNetV2-style) space. There, each block's two activations are independently removable, and the width multiplier varies so that the benchmark spans the small-model collapse regime. The second is a DeiT-style space in which the fraction of feed-forward channels bypassing the GELU activation (the "idle ratio" of RePaViT) varies from 0 to 75% across depths and widths from Tiny upward. For a few hundred base architectures per space, we train the full-activation network and several removal variants on CIFAR-100 and ImageNet-16-120 (the NAS-Bench-201 resolution and splits). All variants use a fixed short schedule. This yields paired accuracy drops, which is the quantity we actually want to predict. A smaller set is trained on ImageNet-1K to check transfer.
The third step is analysis. We test whether each proxy ranks removal variants of the same base architecture correctly, and separately whether it ranks accuracy drops across bases. The second test is where the RePaViT-style width asymmetry should appear. We also test directly whether ZiCo and Jacobian-based trainability scores increase under linearization.
Baselines
The naive reference point is doing nothing: keep every activation and fold only what BaN-OFF can already fold. Every method's accuracy–latency trade-off is plotted relative to this point on the same device. The second naive point is random removal sets matched in measured latency gain. A proxy that cannot beat random selection at equal latency is useless.
The strongest training-free competitors are the best existing proxies applied directly to the linearized training graph. We use ZiCo, which reports Kendall's τ of 0.61 on NATS-Bench TSS (ZiCo), and AZ-NAS, which reports τ = 0.710 on ImageNet-16-120 in NAS-Bench-201 versus 0.584 for ZiCo (AZ-NAS). AZ-NAS is the stronger of the two and is the main comparison. Note that neither number concerns linearized variants: we found no reported result on Lacuna for any zero-cost proxy's rank correlation with the accuracy cost of activation removal, so these baselines must be run by us. Each proxy uses the same minibatches, seeds, and initialization scheme as our score. We also run HaeNAS-style Jacobian covariance, since that pipeline already pairs a proxy with hardware latency. Beating these baselines means higher Kendall τ with the measured accuracy drop, both within and across base architectures, and higher accuracy of the selected removal sets at matched measured latency.
The strongest training-based competitor is DepthShrinker. It learns which activations to remove with differentiable masks and then fine-tunes. It reports 3.06% higher ImageNet accuracy and 1.53× higher V100 throughput than MetaPruning. We run it in our MobileNetV2 space at the same latency targets and with the same final fine-tuning budget as our selected networks. Beating it outright is not required. Our claim holds if we match its accuracy within a small margin at equal latency while spending orders of magnitude less selection compute.
Two ablations isolate the key ingredients. The first replaces the shared-weight difference $\Delta_E$ with the absolute expressivity of the linearized network. This tests whether the paired difference is what matters. The second replaces measured collapsed-graph latency with FLOPs of the collapsed graph. This tests whether measurement matters, since merged dense layers often raise FLOPs while cutting runtime.
Why not just…?
Why not just score the linearized network with AZ-NAS or ZiCo and measure latency on the collapsed graph? This is the most serious threat. It shares our latency half, and differs only in using an absolute proxy instead of a paired expressivity gap. Our reason to expect it to fail is the gradient-noise effect described above, which predicts that ZiCo-like scores rise as activations are removed. There is also a scale problem: absolute scores are dominated by width and depth, which is why earlier proxies struggled to beat parameter counts, rather than by the marginal contribution of a few activations. The within-base ranking experiment settles this directly. If AZ-NAS on the linearized graph matches $\Delta_E$ there, the proposal reduces to "measure collapsed latency and use AZ-NAS," and we would report that as the finding.
Why not just use parameter count or FLOPs? Removing an activation changes neither parameters nor FLOPs of the training graph. After collapse, parameters and FLOPs often increase, because merged dense kernels replace depthwise ones. Yet those merged networks run faster, which is exactly the discrepancy documented by ShuffleNet V2's memory-access guidelines (ShuffleNet V2) and by HW-NAS-Bench. Counting proxies therefore carry no signal about which activations to drop, and the collapsed-FLOPs ablation quantifies how badly they mislead on latency.
Why not just use a latency lookup table or a learned latency predictor? Block-additive tables built on the training graph count three separate convolutions where the deployed graph has one, and they miss compiler fusions (HW-NAS-Bench). Building the collapsed graph at initialization and timing it takes seconds per candidate, so exact measurement is cheaper than training a predictor. We would switch to a predictor only for search spaces too large to profile.
Why not just run DepthShrinker's trained mask search? It works, but it requires training the masks and then fine-tuning for each target network and each latency budget. It also cannot say in advance whether a small model will collapse, which is the failure RePaViT reports on DeiT-Tiny. If DepthShrinker beats our selections by a wide margin at equal latency, the honest outcome is to position $\Delta_E$ as a cheap pre-filter that narrows its search space, and we will report that comparison as measured.
Proposed Experiments
The first decisive check is ranking quality on the trained linearization benchmark. We report Kendall τ between each proxy and measured accuracy drop on CIFAR-100 and ImageNet-16-120, separately within base architectures and across them. The prediction is that $\Delta_E$ exceeds absolute AZ-NAS and ZiCo scores within bases, and that ZiCo is negatively correlated there.
The second check targets the small-model collapse regime. On the DeiT-style space, we ask whether $\Delta_E$, computed at initialization, separates widths that tolerate a 75% idle ratio from those that collapse, reproducing the Tiny-versus-Large asymmetry before any training.
The third check is end-to-end selection on ImageNet-1K in the MobileNetV2 space. We pick removal sets at two measured latency targets per device, fine-tune them with the DepthShrinker budget, and compare Top-1 accuracy at equal latency against random selection, absolute-proxy selection, and DepthShrinker. We also report whether the measured-latency ranking of candidate sets differs between GPU and edge device, which tests whether the selection must be device-specific.
A negative result on the first check, where absolute proxies already rank linearized variants well, would itself be useful. It would mean the missing piece in hardware-aware zero-cost search was only exact collapsed-graph latency, and the whole pipeline would be simpler than we propose.
References
- RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers
- DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks
- Double-Win NAS: Towards Deep-to-Shallow Transformable Neural Architecture Search for Intelligent Embedded Systems
- RepVGG: Making VGG-Style ConvNets Great Again
- To Fold or Not to Fold: a Necessary and Sufficient Condition on Batch-Normalization Layers Folding
- ZiCo: Zero-shot NAS via inverse Coefficient of Variation on Gradients
- AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search
- Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics
- HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
- HaeNAS: Hardware-Aware Efficient Neural Architecture Search via Zero-Cost Proxy
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design