Which Way to Stitch Depends on the Chip

Model stitching joins the early layers of one pretrained network to the later layers of another through a tiny learned adapter, so a handful of off-the-shelf models can yield a whole family of in-between models without training a new network from scratch. Two recent systems give opposite rules for how to do this. One says to put the small, fast model first and the big model second. The other says to put the big model first and the small model second. Both systems judge cost by operation counts or parameter counts, never by time measured on a real device.

We argue that the disagreement is not really about representations. It is about hardware. Where a stitched network spends its time and memory depends on the device: memory-starved microcontrollers pay most for the large early activations, while GPUs pay most for poorly shaped wide layers. So the best stitching direction, and the best place to switch models, should change from chip to chip. We propose to build stitched model families in which both directions are allowed. We also allow two switch points instead of one, for example big-small-big. Adapter training and path selection would be driven by latency and peak memory measured on each target device. The test is whether this larger, measured search space produces better accuracy-versus-latency trade-offs than simply re-ranking an existing stitched family by measured speed.

Core Claim

Stitchable Neural Networks (SN-Net) connects pretrained DeiT anchors with 1×1 convolutions. It enforces a "Fast-to-Slow" direction on the grounds that capacity should grow with depth, and it reports its trade-off curve in FLOPs. ReStNet keeps the donor models frozen, picks the switch point by representation similarity, and argues for "Slow-to-Fast." On CIFAR-10 it reports 95.09% accuracy for Slow-to-Fast against 92.00% for Fast-to-Slow. ReStNet also names measured latency and energy as future work. Efficient Stitchable Task Adaptation (ESTA) freezes the anchors, adds low-rank adapters and a separate bias for each stitch, and samples promising stitches by accuracy per FLOP.

None of these compares directions at matched measured cost on a real device. That gap matters because HW-NAS-Bench finds that the rank agreement (Kendall correlation) between FLOPs and measured latency is as low as 0.18 on the Edge TPU, and below 0.5 across several devices.

Our claim is falsifiable. Once cost is measured on the device, three things should hold. First, which direction is Pareto-optimal (gives the best accuracy at a given latency) will differ across devices. Second, paths with two switch points will fill regions of the measured Pareto front that no single-switch path reaches. Third, both effects will survive the cheap alternative of re-ranking SN-Net's existing stitches by measured latency. If the measured fronts of the two directions coincide once cost is matched, the directionality debate was an artifact of counting FLOPs. That outcome is publishable too, but it would kill the method.

Research Program

We take two or three pretrained anchors from one family and freeze them. The CNN pool is ResNet-18, ResNet-50 and a MobileNet. The transformer pool is DeiT-Tiny, DeiT-Small and DeiT-Base, which lets us compare directly with SN-Net. The candidate space contains every single-switch path in both directions, plus two-switch paths restricted to a coarse grid of block boundaries so the space stays in the low thousands. Each junction gets a 1×1 convolution or linear adapter, initialized by least squares on about 100 images as in SN-Net. Training follows ESTA: shared low-rank adapters inside the anchors plus a separate bias for each stitch.

Hardware awareness enters in two places.

The first is cost measurement. We compile and time a stratified sample of several hundred paths on each device. The junction between two anchors blocks the operator fusion that edge compilers normally perform, so adding up per-layer latencies from a lookup table is suspect, as HW-NAS-Bench notes for lookup-table cost models in general. We therefore fit a per-device predictor in the style of LitePred or HELP. We add junction features to it: the anchor pair, the tensor shape at the junction, and the switch count. We then report how much these features improve its rank correlation over a plain additive lookup table.

The second is training-time sampling. During training we sample stitches in proportion to how close their predicted device cost and current accuracy place them to the device's Pareto front. This replaces ESTA's accuracy-per-FLOP saliency. Doing so yields one set of adapters per device, trained in a few GPU hours rather than one set for all devices.

Because a two-switch path gives the adapters more freedom to fit, we also run a cheap validity check. It rests on the finding in Functional Alignment Can Mislead and Invariance-Aware Model Stitching that a stitch can reach high accuracy by exploiting shortcuts that neither donor used. We measure how often the stitched model agrees with its donors on shifted data. Paths that stay accurate in-distribution but break donor agreement out of distribution are flagged. We do not generate metamers, the inputs those papers synthesize to test shared invariances, because producing them per sample does not scale to thousands of paths.

Baselines

The naive reference point is zoo selection: for each latency budget, deploy the most accurate unmodified anchor that fits on that device. It costs no training, and any stitched family must beat it between anchors to justify itself.

The strongest stitching competitor is SN-Net re-ranked by measured latency. We train SN-Net's Fast-to-Slow family with its own recipe on the same anchors and the same ImageNet-1K data. We then time its paths on each device and keep its measured Pareto front. It is the strongest competitor because it is exactly what a practitioner would do first. SN-Net reports 81.5% top-1 at 11.2G FLOPs for about 110 GPU hours. No reported result found on Lacuna for SN-Net's measured on-device latency. We also run ESTA with its FLOP-based sampling, matching our trainable-parameter count and GPU-hour budget. ESTA reports producing deployable models in 5 GPU hours with 4.6M trainable parameters.

The strongest non-stitching competitor is Once-for-All (OFA), a single supernet trained once and specialized per device with latency lookup tables. It reports 80.0% ImageNet top-1 under 600M MACs, at roughly 1,200 V100 GPU hours. We use its released supernet and search it with our measured predictors rather than retraining it. The comparison is therefore about Pareto quality for the compute actually spent, and OFA's upfront cost is reported alongside.

Three ablations isolate our ingredients. The first uses the same bidirectional, two-switch space but samples by FLOPs; it isolates the effect of hardware-driven sampling. The second uses measured sampling but only one direction; it isolates the effect of direction. The third uses measured sampling in both directions but a single switch; it isolates the effect of the second switch point.

Beating a baseline means a higher hypervolume, meaning a larger area under the device's accuracy-versus-measured-latency front, across three training seeds. On at least two of the devices, it also means at least a 0.5-point top-1 gain at matched latency over latency-re-ranked SN-Net.

Why not just…?

Why not just re-rank SN-Net's stitches by measured latency? This is the real threat. If SN-Net's candidate set already contains the device-optimal paths, measuring latency is only bookkeeping and our method adds nothing. It is the primary baseline for that reason. The first two ablations separate the two possible sources of any gain: a larger candidate space versus better sampling. If both lose to re-ranking, we will report that measured-latency re-ranking is sufficient.

Why not just use Once-for-All with a latency table? OFA gives excellent fronts, but only within one backbone. Adding a new family such as transformers means paying the roughly 1,200-GPU-hour training again. OFA's own paper and Does Interference Exist When Training a Once-For-All Network? also discuss the interference that arises when many sub-networks share one set of weights. We compare fronts per GPU-hour. OFA may well win on absolute accuracy for CNNs on some devices, and we would report that.

Why not just prune or quantize one strong model per device? Tools like HAQ produce one static model per budget. A stitched family instead lets the device switch paths at runtime as its budget changes, with no reloading of new weights. Quantization is also complementary rather than competing, so we quantize all methods identically for the Edge TPU and the microcontroller.

Proposed Experiments

The ImageNet-1K experiments use the DeiT and CNN anchor pools on the Jetson TX2, Raspberry Pi 4 and Pixel 3, the edge devices profiled in HW-NAS-Bench, with Edge TPU latency but no energy, since HW-NAS-Bench notes the board lacks accessible power rails. We add CIFAR-10 and ImageNette to reproduce ReStNet's 95.09% versus 92.00% direction comparison and then re-plot it against measured latency and peak memory. The microcontroller regime uses small CNN anchors on GAP8 with Visual Wake Words, measuring latency and peak memory in the chip's L2 buffer. The reference there is DUCCIO, which reports an 87.4% memory reduction and a 54.2% latency reduction over its seed model on Visual Wake Words.

Three checks decide the question. First, the Pareto-optimal direction must differ across at least two devices at matched measured latency; if it does not, the core claim fails. Second, the bidirectional two-switch space must beat SN-Net re-ranked by measured latency in front hypervolume. Third, paths flagged by the donor-agreement check must be reported, so that no front gain comes from shortcut-exploiting stitches.