Ridge-Point Straddling: A Physical Account of When Hardware-Aware Architectures Stop Transferring

Neural networks that are fast on one chip are often slow on another, and nobody can say in advance which pairs of models will swap places. A model found by a latency-driven search on an edge GPU ran about 50% faster there than a model searched for a Raspberry Pi, yet the ordering can reverse on the Pi. Operation counts (FLOPs) barely predict speed at all: their rank correlation with measured latency falls to 0.18 on Google's Edge TPU (HW-NAS-Bench). Current fixes are statistical. One is a latency predictor retrained with a few dozen measurements per new device (One Proxy Device Is Enough). Another transfers kernel-level predictors across a pool of 85 platforms (LitePred). These tools track latency, but they cannot tell an architect why a ranking flips or which models are safe to deploy across several chips.

We propose a simple physical explanation and test it at scale. Every chip has a ridge point: the ratio of its peak arithmetic rate to its memory bandwidth. It marks how many operations per byte of memory traffic (the arithmetic intensity) a layer needs before the chip's arithmetic units, rather than memory, become the bottleneck. Our hypothesis is that two architectures swap latency rank between devices A and B mainly when a large share of their work consists of layers whose intensity falls between the two devices' ridge points. Those layers are compute-limited on one chip and memory-limited on the other. A third ingredient accounts for the tiny, fragmented networks common in search benchmarks: a fixed per-kernel launch cost. If the hypothesis holds, a device can be summarized by three numbers measured with a handful of synthetic microbenchmarks, with no architecture samples needed. That summary would predict cross-device rank inversions, and it would support a search for architectures whose speed carries over across a set of target chips.

Core Claim

For device $d$ and architecture $a$ with layers $l$, we model latency as

$$ \hat{L}_d(a) = \sum_{l \in a} \max\!\left(\frac{F_l}{P_d}, \frac{B_l}{W_d}\right) + K(a)\, c_d , $$

where $F_l$ is the layer's arithmetic operation count, $B_l$ its bytes moved to and from off-chip memory, $P_d$ and $W_d$ the device's effective (not datasheet) compute rate and bandwidth, $K(a)$ the number of launched kernels after a stated fusion rule, and $c_d$ a per-kernel fixed cost. The max term is the speed-of-light latency of The Sparsity Roofline, applied per layer. The overhead term captures the fragmentation and launch costs that ShuffleNet V2 identified qualitatively. We do not claim that this three-parameter model predicts absolute latency better than learned predictors. The claim is narrower and falsifiable in two parts. First, the straddle mass of an architecture pair should predict whether their order flips between A and B. We define straddle mass as the fraction of their latency difference carried by layers with intensity between the two ridge points $P_A/W_A$ and $P_B/W_B$. Second, the model, calibrated only from microbenchmarks, should give zero-shot cross-device ranking good enough to beat FLOPs and memory-traffic proxies by a wide margin, and should approach few-shot statistical transfer at matched measurement budgets.

This resolves a tension in the literature. One Proxy Device Is Enough reports latency rank correlations above 0.98 within a hardware family and 0.6–0.8 across families. HW-NAS-Bench reports correlations below 0.5 across heterogeneous devices. On our account these results are consistent. Devices in the same family have nearly the same ridge point, so almost nothing straddles. Heterogeneous devices have far-apart ridge points and large straddle mass.

Relation to Prior Work

Three papers motivate the proposal by showing the gap without explaining it. HW-NAS-Bench measures FLOPs–latency breakdown and search overfitting on six devices, but offers no mechanism. HURRICANE finds a 24× latency spread among operators with identical FLOPs on a VPU and responds by profiling operators on each device. One Proxy Device documents monotonicity and patches its failure with linear adaptation. We keep their data and replace the black box with a mechanism that can be checked.

Three papers supply the method. The speed-of-light formula comes from The Sparsity Roofline, which applied it to fixed sparse models rather than to search or cross-device ranking. How to keep pushing ML accelerator performance? Know your rooflines! supplies the point that effective ceilings sit below the datasheet roof and differ by memory level. That is why we fit $P_d$ and $W_d$ from microbenchmarks rather than reading them from spec sheets. HarDNet supplies a byte-counting proxy (convolutional input/output traffic). It also notes that on small networks launch overhead swamps traffic savings, which is why the model includes $c_d$.

The closest existing work is Searching for Fast Model Families on Datacenter Accelerators. It used roofline reasoning to design a search space, replacing depthwise with fused convolutions to raise intensity, and scaled models against measured latency on V100 and TPUs. Its models stayed target-specific: GPUs favored ReLU, while TPUs mixed ReLU and Swish. It made no prediction about when rankings transfer between chips. We differ in three ways. We use the roofline as a predictive model of cross-device rank inversion rather than as a design heuristic. We evaluate on six heterogeneous edge devices rather than two datacenter accelerators. And we turn the model into an objective for portability across several chips.

Can Tensor Cores Benefit Memory-Bound Kernels? (No!) constrains the design. When a chip's ridge point moves right, memory-limited kernels gain at most a bounded speedup, and forcing them onto specialized units adds a "reshaping tax." This warns that some slowdowns come from data-layout penalties outside any roofline, and we track those as residuals.

Research Program

The first component is calibration. For each device we run a small fixed microbenchmark suite: a dense matrix multiply swept over sizes, a streaming elementwise copy, and a chain of tiny kernels. We fit $P_d$, $W_d$, $c_d$ by least squares on these synthetic kernels only, so that no architecture from the evaluation space ever touches calibration. For the two devices in HW-NAS-Bench whose numbers are simulated or estimated, the Eyeriss simulator and the ZC706 FPGA model, we run the same kernels through the same estimator.

The second component is the analytic model. For each architecture we compute $F_l$, $B_l$ and $K(a)$ under two explicit fusion rules. Under the unfused rule, every operation reads and writes DRAM. Under the fused rule, elementwise operations and normalization fold into the preceding convolution. We choose the rule per device from the calibration fit, never from evaluation data. This directly probes the fusion gap that block-additive benchmarks leave open.

The third component is analysis. For every device pair and a large sample of architecture pairs, we compute straddle mass and record whether the measured order flips.

The fourth component is search. We run an evolutionary search whose latency term is the worst-case normalized $\hat{L}_d$ across a set of target devices. The worst case pushes architectures toward layers that sit above every target's ridge point, or at least not between them. We compare the resulting portability penalty against single-device search.

Baselines

The naive reference is FLOPs, which HW-NAS-Bench reports at a Kendall correlation of 0.18 with latency on the Edge TPU and below 0.5 on several devices. It is the default proxy and the floor any physical model must clear. The second baseline is scalar memory-traffic counting, using convolutional input/output traffic as in HarDNet, plus memory access cost and kernel count as in ShuffleNet V2. It uses the same byte counts as our model, but without device-specific ridge points. Computed identically, it tests whether the device-dependent max operation matters at all. On the benchmarks in the file we found no reported rank-correlation result for this proxy.

The strongest competitor is AdaProxy from One Proxy Device Is Enough. It fits a latency predictor on one proxy device and linearly adapts it with 30–80 target measurements. The paper reports rank correlation rising from 0.6–0.8 before adaptation to above 0.9 "in almost all tested scenarios." We will run it on the HW-NAS-Bench NAS-Bench-201 space with a Jetson TX2 proxy, using the same random target samples at budgets of 0, 10, 30 and 80. Our model receives an equivalent microbenchmark budget, counted as the same number of on-device runs, and a hybrid variant may additionally use the same architecture samples. For LitePred, the paper reports 99.3% average prediction accuracy across 85 platforms. We will compare against it only on devices and spaces where its code and data are available. That number measures absolute accuracy, not rank inversion, so it is not directly comparable to our main metric.

We would count the proposal as beating the baselines on three conditions. At zero architecture samples, our model must exceed FLOPs and memory traffic by at least 0.2 in Spearman correlation on cross-family pairs. At 30 samples, the hybrid must match or exceed AdaProxy. And on inversion prediction, which neither baseline attempts directly, straddle mass must achieve an area under the ROC curve well above that of FLOP-difference or traffic-difference predictors.

Three ablations isolate the key ingredient. Replacing the max with a sum removes the regime switch. Dropping $c_d$ removes the overhead term. Swapping the fitted $P_d, W_d$ for datasheet values tests whether effective ceilings matter.

Why not just…?

Why not just measure latency on each target device? For a fixed benchmark space this is the ground truth, and we use HW-NAS-Bench measurements for exactly that purpose. It does not scale to many devices or to open search spaces, as One Proxy Device and LitePred both stress. MO-HDNAS reports that a measurement-heavy baseline needed 20.87 GPU hours against its own 0.65 (MO-HDNAS). More importantly, measurement says nothing about why a model fails to port. The HW-NAS-Bench result that the edge-GPU model was about 50% faster on its own device is exactly what our portability search is designed to reduce. We will report whether it does.

Why not just use AdaProxy or LitePred? This is the real threat. With 30–80 samples, AdaProxy already reaches rank correlation above 0.9, and our model may not beat that in the few-shot regime. We therefore do not stake the proposal on few-shot accuracy. The measured claims are three. The model should work at zero architecture samples. It should predict which pairs invert. And adding roofline features to AdaProxy should help most on cross-family pairs with high straddle mass. If AdaProxy's residual errors show no relation to straddle mass, the mechanism adds nothing beyond interpretability, and we will say so.

Why not just count memory traffic? A scalar byte count ignores the device, so it cannot distinguish a chip with abundant bandwidth from one starved of it. By construction it predicts identical rankings on every device, which HW-NAS-Bench shows is false. The ablation that keeps our byte counts but drops the per-device max tests this directly. If the scalar proxy matches our model, the ridge-point story is wrong.

Why not just add hand-picked roofline features to a gradient-boosted predictor? This is plausible and cheap, and we include it as a baseline. We give it layer intensities and kernel counts, train it on the proxy device, and adapt it with the same samples. If it wins, the straddle mechanism still stands as the explanation of its features, but the model's value would be as a feature design rather than a predictor. The inversion analysis remains the distinguishing contribution.

Proposed Experiments

The data are HW-NAS-Bench: 15,625 NAS-Bench-201 cells on CIFAR-10, CIFAR-100 and ImageNet-16-120, plus the FBNet layer-wise space, on six devices. These are the Jetson TX2, Raspberry Pi 4, Pixel 3, Edge TPU, the simulated Eyeriss and the ZC706 FPGA. Accuracy comes from NAS-Bench-201, whose best cells reach 94.37%, 73.51% and 47.31% on the three datasets. We report Spearman and Kendall correlations per device pair, inversion-prediction ROC-AUC as a function of straddle mass, and a calibration plot of inversion probability against straddle-mass bins. For search, we report the Pareto front of accuracy against worst-case latency over device sets, together with the portability penalty relative to per-device ProxylessNAS-style search.

The decisive check comes first. Inversion probability should rise steeply with straddle mass and be near zero for pairs whose layers all lie on the same side of both ridge points. If the curve is flat, the hypothesis fails. The second check is zero-shot rank correlation from microbenchmark calibration alone, compared with FLOPs, which reaches 0.18 on the Edge TPU, across all device pairs. The third check is failure analysis. Residual inversions not explained by straddle mass will be examined for operator fallback of the kind HURRICANE found with squeeze-and-excitation blocks on the VPU, and for the layout penalties described in the Tensor Core study. Each residual is labeled as a case outside the roofline rather than absorbed into a fudge factor.

Two limitations are known in advance. First, NAS-Bench-201 networks are small, so the overhead term may dominate on some devices; the FBNet space provides a regime with more memory traffic. Second, the FBNet latencies are block-additive, so fusion effects there can only be tested on the devices we can re-measure directly.