Co-Searching the Input Format and the Network Under End-to-End Latency

Most image models deployed on edge devices receive JPEG files. JPEG was designed for human eyes: it throws away fine detail people do not notice, and it spends bits on smooth textures people do notice. Before the network runs, the device must also decode that file back into a full pixel grid and copy it to the accelerator. Two research communities each handle half of this pipeline. Machine-oriented compression redesigns the stored format and leaves the network fixed. Hardware-aware neural architecture search, an automated way to choose layer types and widths under a measured speed budget on a target chip, redesigns the network and assumes the input arrives as free RGB pixels. We propose to search both at once. One search would choose which frequency components of each image are stored and fed to the network, and also the network that consumes them. The objective would be wall-clock time measured from stored bytes to prediction on the real device.

The case for joint search rests on one observation: the gap between counted arithmetic operations (FLOPs) and measured speed applies to the input pipeline just as it applies inside the network. Keeping 24 frequency channels instead of 32 saves decode work and bytes. It can still run slower if it breaks the channel alignment the accelerator's matrix units expect. And which frequencies matter for accuracy depends on the network reading them. So a format chosen for a fixed network, followed by a network searched for a fixed format, may land on a worse speed-accuracy tradeoff than choosing the two together. The proposal is built so that this hypothesis can fail cleanly.

Core Claim

For a given device and latency budget, the best pairing of input representation and backbone differs from the best pairing found by optimizing each in turn. It also differs from simply lowering the image resolution. We expect three signatures if the claim is true. First, the retained frequency sets will not be pure low-pass, so they cannot be reproduced by downsampling. Second, the retained frequency sets will differ across devices for the same task. Third, the stem and early stages of the searched network will change with the chosen representation.

Formally, let $r$ be an input representation: which DCT bands are kept for each of the Y, Cb, Cr channels, their quantization steps, and the chroma subsampling. A DCT, or discrete cosine transform, is the per-8×8-block frequency decomposition JPEG uses. Let $a$ be a backbone architecture and $h$ a target device. We solve

$$ \min_{r,a}\ \mathrm{Err}(r,a) \quad \text{s.t.} \quad T_{\text{decode}}(r,h) + T_{\text{transfer}}(r,h) + T_{\text{forward}}(a,r,h) \le B, $$

where $T_{\text{decode}}$ is the time to read and entropy-decode the stored bytes on the device CPU, $T_{\text{transfer}}$ is the time to move the result to the accelerator, $T_{\text{forward}}$ is the compiled forward-pass time, and $B$ is the latency budget. The objective is measured as one pipeline, never as a sum of separately profiled parts. Measuring it this way is part of the contribution.

Relation to Prior Work

Three papers motivate the gap. The Image Calculator builds an automated design space over color conversion, subsampling, DCT coefficient selection, quantization and lossless coding, using analytical models of accuracy and latency. It shows that I/O, CPU decompression and PCIe copies can dominate inference. However, it keeps the downstream network (for example Swin or MaxViT) frozen. We search the network jointly, and we treat that frozen-network pipeline as our main baseline. HW-NAS-Bench measures latency for tens of thousands of architectures on six devices and finds FLOPs-latency rank correlation as low as 0.18 on the Edge TPU. Every one of its measurements assumes decoded RGB input, so ingestion cost lies outside its scope. JMSNAS jointly searches an MBConv network and its split points using profiled latencies, but transmits uncompressed or conventionally compressed tensors. In short, prior work searches the format or the network, but not both under one measured clock.

A second group supplies evidence about which frequencies machines need. DeepN-JPEG derives quantization tables from DNN frequency sensitivity rather than human vision. It reports about 3.5× higher compression than JPEG at equal ImageNet accuracy. Task-Aware Quantization Network predicts per-image JPEG-compatible tables through a differentiable proxy for entropy coding. Both target fixed classifiers and never measure accelerator latency. We reuse their finding that useful frequencies are not simply the lowest ones. That finding is what separates our search space from resolution reduction.

A third group constrains the design. Deep Residual Learning in the JPEG Transform Domain maps ResNet operations exactly into the DCT domain. The resulting weight expansion and sparse operations lack mobile runtime support. We therefore do not port spatial convolutions into the DCT domain. Instead, retained bands become ordinary dense input channels at block resolution, consumed by a searchable stem. ShuffleNet V2, All You Need Is a Few Shifts and Interleaved Group Convolutions show that memory access and fragmentation, rather than FLOPs, govern speed. This is why the representation choices are quantized to hardware-friendly channel groupings and why the objective is measured rather than counted.

Methods are borrowed from the HW-NAS literature: weight-sharing supernets and differentiable search from FBNet, multi-objective evolutionary search from MoGA, and learned latency predictors from LitePred and On Latency Predictors for Neural Architecture Search.

Research Program

The first component is the representation space. Images are stored offline in a simple block-DCT container. For each 8×8 block, only the selected bands are entropy-coded, using a lightweight coder decodable on an ARM CPU. Unretained bands are never stored, so the decode work shrinks with the selection. Band choices are made in groups of 4 or 8 channels, and the group size itself is a searchable variable. This lets the search discover whether alignment to the accelerator's vector width matters on each device.

The second component is a backbone search space. It is an MBConv-style space as in FBNet and MoGA, extended with a representation-dependent stem. The stem can consume block-resolution frequency channels directly, upsample a subset, or recombine luma and chroma at different stages. One supernet is trained over both representations and architectures, with representations sampled per batch.

The third component is the end-to-end latency predictor. We measure complete pipelines on the device, from stored bytes through decode and copy to the compiled forward pass. These measurements train a predictor whose inputs are the joint configuration. We explicitly compare it against an additive estimate that sums a decode lookup table with a per-block forward lookup table. HW-NAS-Bench already documents that additive tables break under compiler fusion. We expect the break to be worse when decode and transfer overlap with compute.

Search then runs NSGA-II over the joint space using supernet accuracy and predicted latency. The top candidates are retrained from scratch and re-measured on device.

The targets are four devices from HW-NAS-Bench: Jetson TX2 (edge GPU), Raspberry Pi 4 (ARM CPU), Pixel 3 (mobile CPU) and Edge TPU. They span very different memory and decode situations. The Edge TPU's integer-only, static-shape requirements will expose whether alignment effects dominate. We study two regimes. In the local-storage regime, files are read from flash. In the link-constrained regime, files arrive over a fixed-bandwidth link, so stored bytes add transmission time.

Baselines

The naive reference is standard JPEG at several quality settings, decoded with an optimized CPU decoder (libjpeg-turbo) and fed to MobileNet-family networks, with the full pipeline timed on each device. Its job is to show how much end-to-end time the conventional path actually spends outside the network on these devices. No such per-device breakdown is reported in the file.

The strongest existing representation method is the Image Calculator approach. We implement its format search over our representation space, using its accuracy and latency modeling, with the backbone frozen to a fixed searched network. The frozen network is the joint search's own best RGB-trained architecture, so the backbone is not handicapped. On ImageNet, The Image Calculator reports up to 14.2× faster end-to-end inference than JPEG with "less than 1–5%" accuracy loss, and 3.8× lower inference time with 2.5% higher accuracy for Swin and MaxViT. Those numbers come from its own hardware and its frozen transformer models. We do not expect to reproduce them on edge devices. We will report our reimplementation's numbers alongside them.

The strongest existing architecture method is hardware-aware NAS on RGB input in the MoGA/FBNet style, searched under forward-pass latency only, then deployed with real JPEG decode added. MoGA reports 75.9% top-1 for MoGA-A and 75.3% for MoGA-C on ImageNet, measured on mobile GPU without ingestion cost. We use these figures to calibrate our search space against theirs, not as end-to-end targets.

The decisive ablation is sequential search. We first run format search for a fixed network, then run NAS on the chosen format, and iterate once more. A second ablation keeps the joint space but optimizes FLOPs plus bytes instead of measured time. A third uses the additive lookup-table latency instead of the measured end-to-end predictor.

All methods share the same ImageNet training recipe, the same supernet training epochs and the same number of searched candidates. Every method also gets the same retrain-from-scratch budget for its top three candidates. All timings use the same compiler (TensorRT on Jetson, TFLite elsewhere) and the same batch size. The joint method beats the strongest baseline if, at matched measured end-to-end latency on a device, it improves ImageNet top-1. The improvement must exceed three seed standard deviations of the retrained models, which we estimate before search. Equivalently, at matched accuracy, the joint method must lower latency on at least two of the four devices.

Why not just…?

Why not just lower the input resolution and run ordinary HW-NAS? This is the most serious threat. Keeping only the lowest DCT bands is close to downsampling, and downsampling cuts decode, transfer and compute together. We include RGB at 112, 128, 160 and 192 pixels as baselines, each with its own HW-NAS search under end-to-end latency, using hardware JPEG decode where it exists. If these match the joint search, the conclusion is that frequency selection adds nothing beyond resolution, and we will report that. The evidence that it might not match comes from DeepN-JPEG, which found that classifiers depend on high-frequency bands that low-pass removes. No reported result found on Lacuna for a resolution-reduced HW-NAS baseline measured end to end.

Why not just optimize the format and then the network separately? This is the sequential ablation. If coupling is weak, this ablation will tie the joint search, and the joint machinery is not worth its cost. We would see that directly at matched search budget.

Why not just decode on the GPU with a hardware decoder? On Jetson we use nvJPEG for the JPEG baselines, so the conventional path gets its best decoder. Hardware decoding does not reduce stored or transmitted bytes, though. The Raspberry Pi and Pixel CPU paths have no comparable decoder, so this alternative cannot settle the question on those devices. No reported result found on Lacuna for nvJPEG end-to-end edge latency.

Why not just train an RGB HW-NAS model on JPEG-degraded images? Augmentation makes a model robust to missing detail, but it cannot restore bands the encoder discarded. It also leaves decode and transfer cost unchanged. We include it as a variant of the RGB baseline, trained with matched augmentation budget.

Decisive Checks

The program succeeds only if all three outcomes hold. First, joint search must beat both sequential search and the best resolution-reduced RGB baseline at matched measured latency on at least two devices. Second, the selected band sets must differ measurably from low-pass and from each other across devices. Swapping the Raspberry Pi optimum onto the Jetson, or the reverse, should cost accuracy or time. This parallels the roughly 50% speed gap HW-NAS-Bench reports for cross-device transfer of forward-only models. Third, the measured end-to-end predictor must rank held-out pipelines more faithfully than the additive lookup-table estimate. We measure this by Kendall correlation against device timings.

If the first check fails while the second holds, the device-specific format matters but the network does not need to co-adapt. In that case the Image Calculator design, with a frozen network, is sufficient, and that result is still worth publishing. If the third check fails, the cheaper additive tables are adequate for ingestion-aware search, which simplifies the method. As secondary diagnostics, we will report per-stage timing breakdowns and stored bytes per image.