Searching Split-Computing Encoders Against Measured End-to-End Latency

When a phone or camera cannot run a full neural network, a common workaround is split computing: the device runs the first few layers, compresses the intermediate features, sends them over a wireless link, and a server finishes the job. The compression here is task-aware. It is trained to keep whatever the downstream classifier or detector needs, not what a human viewer would notice. The field has a well-known tension, which we will call the bottleneck paradox. Splitting deeper produces features that compress to fewer bits, but it makes the weak device do more work. The field has been resolving this tension with the wrong ruler. Encoders are compared by parameter count, by operation count (FLOPs), or by the product of encoder size and bitrate. Meanwhile, a separate literature has shown that operation counts predict real speed poorly, and that a design that is fast on one chip is often slow on another.

This proposal measures real end-to-end latency for split encoders: device compute, including quantization and entropy coding, plus transmission time, plus server time. It then searches over encoder architecture, split depth, and bottleneck shape against a hard limit on that measured latency, separately for each device and each bandwidth. The claim has two parts, and it can fail. First, the rankings of split encoders produced by current proxies disagree substantially with their measured-latency rankings on at least some edge hardware. Second, where they disagree, a hardware-aware search with a hard constraint finds encoders that are more accurate at a given latency budget than the best hand-designed encoder chosen with an equally careful measured-latency sweep. If the first part fails, the second is unnecessary, and the study says so.

What exists and what does not

Three lines of work each touch two of the three ingredients.

The first is split computing without automated design. Towards Collaborative Intelligence Friendly Architectures for Deep Learning inserts a hand-designed pair of 1×1 convolutions at the split point: one on the device shrinks the channels, and one on the server restores them. They measure real latency on a Jetson TX2 over 3G, 4G and Wi-Fi, and report up to 77x latency reduction on 3G with under 2% accuracy loss. Their bottleneck shape and front-end are fixed by hand, though. SC2 Benchmark compares ten supervised compression methods across 180 models. It identifies the Entropic Student, which distills a teacher's intermediate features into a small encoder and then fine-tunes the downstream part with the encoder frozen, as the strongest. That encoder is about 40x smaller than a standard learned image compressor. The authors state that device cost was measured by parameter count and model size, not latency. Rate-Distortion in Image Coding for Machines proves that deeper features need fewer bits for a fixed task loss, which they call the "deeper is better" rule. They report −50.55% BD-rate (bitrate savings at equal accuracy) on COCO by matching YOLOv3's layer 108. They say nothing about what running 108 layers costs on an edge chip. These papers motivate the proposal: they establish the paradox but measure only one side of it.

The second is search for split models without compression. JMSNAS jointly searches MobileNetV2-style blocks and the partition point across edge tiers. It reports up to 20.1% lower latency than full cloud offload. However, it transmits uncompressed feature tensors, and it estimates latency by summing per-operation profiler timings. This is the closest existing work. We differ in three ways. The bottleneck and entropy coder are inside the search. Latency is measured on the whole compiled encoder rather than summed per operation. And the constraint is hard rather than folded into the loss.

The third is hardware-aware search on a single device. HW-NAS-Bench reports a Kendall rank correlation between FLOPs and latency as low as 0.18 on Edge TPU. It also finds cross-device latency correlations frequently below 0.5. ShuffleNet V2 explains part of the gap through memory traffic. Unequal input and output channel widths, element-wise operations, and branching all cost time without costing FLOPs. This is directly relevant here, because a channel-reducing bottleneck such as 256→1 is exactly an unequal-width projection. ConNAS supplies the search method we reuse. When a candidate violates a constraint, it bends the architecture gradient toward the feasible region instead of adding a tuned penalty term. It reports 100% constraint satisfaction and results within 0.14 to 0.65% of the best feasible architecture on NATS-Bench. All three papers consider standalone models with no transmission term. Overcoming the Latency Bottleneck in Learned Compression and Scalable Image Coding for Humans and Machines constrain the design: sequential entropy models and multi-layer codecs are slow, the latter roughly 3x slower to decode. Entropy-model choice must therefore be a searched, measured variable, not a free add-on.

Core Claim

End-to-end split latency for an encoder architecture $a$ on device $h$ at bandwidth $B$ is

$$T(a,h,B) = T_{\text{dev}}(a,h) + \frac{R(a)}{B} + T_{\text{srv}}(a).$$

Here $T_{\text{dev}}$ is measured device time including quantization and entropy coding, $R(a)$ is the bits per sample the encoder emits at its operating point, and $T_{\text{srv}}$ is server decode-and-inference time. The search solves

$$\max_a \ \text{Acc}(a) \quad \text{subject to} \quad T(a,h,B) \le \tau, \ \ M_{\text{dev}}(a,h) \le m,$$

where $\tau$ is the latency budget and $M_{\text{dev}}$ is peak device memory with limit $m$.

The prediction is a regime map, not a uniform win. At low bandwidth the transmission term dominates, so a proxy that tracks bitrate should be nearly sufficient and search should help little. At high bandwidth, or on accelerators where FLOPs track latency poorly such as Edge TPU, $T_{\text{dev}}$ dominates and proxy-chosen encoders should be measurably off the frontier. Locating the boundary between these regimes is itself a result practitioners can use.

Research Program

The first stage is a measurement study, and it decides whether the rest is worth doing. Using sc2bench, we take the SC2 baseline encoders. We add a structured sweep of Entropic-Student-style encoders over split depth, bottleneck channel count, and entropy model (factorized versus hyperprior). We compile each for Jetson TX2, Raspberry Pi 4, and Edge TPU, three of the HW-NAS-Bench devices, and time the full device pipeline through bitstream output. For each device we compute Kendall tau between measured $T_{\text{dev}}$ and each of parameter count, FLOPs, the encoder-size-times-rate metric, and an additive per-layer lookup-table estimate. We also record how much of $T_{\text{dev}}$ goes to entropy coding. We expect this share to be large for small encoders, which every proxy ignores.

The second stage is the search. The device-side search space combines MBConv and ShuffleNet V2 units, with split depth, bottleneck width, bottleneck kernel, quantization bit-width, and entropy model as searched choices. The server side is a fixed frozen teacher tail, ResNet-50 for ImageNet and Faster R-CNN as in SC2 for COCO, so that accuracy differences come from the device side. Supernet training uses the Entropic Student's distillation loss plus a rate term. Device latency comes from a predictor trained on whole compiled subgraphs rather than summed layers, and we re-measure on hardware every few epochs to correct predictor drift. The hard constraint uses ConNAS gradient modification on $T$, not a penalty weight. We run the search for each pair of device and bandwidth, using 3G, 4G and Wi-Fi throughput levels following Eshratifar et al. The final models are retrained from scratch and measured on the device.

Baselines

The naive reference point is full offload with on-device JPEG or WebP input compression, timed on the same devices using their hardware or library encoders. SC2 reports that this collapses in accuracy at low rate. At high bandwidth, however, it may well win on latency, and it must be on every plot.

The strongest existing method is the Entropic Student from SC2 Benchmark. We give it the same treatment as our method: the full sweep over split depth, bottleneck width and entropy model, all measured on device, with the best configuration for each device and bandwidth selected by measured latency. It gets the same training recipe, the same total GPU-hours as the search, and the same teacher. The file reports its encoder as roughly 40x smaller than a mean-scale hyperprior. However, no reported result was found on Lacuna for the Entropic Student's ImageNet accuracy at a specific bitrate or latency, so we reproduce it with sc2bench. Beating it means at least one point of top-1 accuracy, or of COCO mAP, at equal measured end-to-end latency on at least two of three devices in the device-dominated regime. This is evaluated over three seeds, with confidence intervals that exclude zero.

The JMSNAS-style baseline reproduces joint block and split search with summed per-operation latency, plus a fixed Eshratifar bottleneck added after the search. JMSNAS reports up to 20.1% latency reduction over cloud offload, but that comparison involved no compression, so it is not directly comparable here.

Two ablations isolate the key ingredients. The first keeps our search but swaps measured whole-graph latency for FLOPs and for the additive lookup table. This tests whether the measured latency is what matters. The second swaps the hard constraint for a tuned soft penalty with a matched tuning budget. This tests whether ConNAS's 100% constraint satisfaction carries over to an objective that includes a transmission term.

Why not just…?

Why not just compress the input image with the device's hardware JPEG encoder and offload everything? SC2 shows input compression losing badly at low bitrate, and Scalable Image Coding for Humans and Machines reports 37 to 80% bitrate savings for task-aware latents over HEVC and VVC on COCO. That evidence concerns bits, not wall-clock time. With fast links and a dedicated codec chip, offload can win, and our regime map will report exactly where it does.

Why not just sweep a hand-designed encoder family with measured latency? This is the real threat, and it is why the sweep baseline is given equal compute and on-device selection. ConNAS's 1.55% gain over hand-designed models comes from a different setting, a microcontroller budget, so it does not settle the question. If the sweep matches the search, the contribution shrinks to the measurement study and the regime map, which we would report as the finding.

Why not just rank designs by FLOPs or by encoder size times rate? HW-NAS-Bench's Kendall tau of 0.18 on Edge TPU, and ShuffleNet V2's 58% speed gap at equal FLOPs, suggest these proxies will fail. Both results come from standalone models, however. Stage one tests the claim directly on split encoders, and in the bandwidth-dominated regime we expect the proxies to be adequate.

Why not just add a bottleneck to JMSNAS? That is the JMSNAS-style baseline above. If it matches our method, then joint search of the bottleneck is unnecessary.

Decisive Checks

The project lives or dies on two numbers. The first is the stage-one Kendall tau between proxies and measured device latency for split encoders on each device. If it is above roughly 0.8 everywhere, measured latency adds little and we stop after publishing the measurements. The second is the accuracy gap at matched measured latency between the searched encoders and the measured-latency sweep of the Entropic Student, in the high-bandwidth and Edge TPU settings where the gap should be largest. A further supporting check is whether the searched split depth moves shallower as bandwidth rises and as device latency per operation worsens. That pattern would be the hardware-dependent qualification of the "deeper is better" rule that this project sets out to measure.