Small devices such as cameras, drones and wearables increasingly carry an FPGA (a chip whose logic can be rewired after manufacture) to run neural networks at low power. A useful trick on such chips is the early exit: a small classifier attached partway through the network stops computation when it is already confident. When that classifier is not confident, today's FPGA designs simply keep computing locally. Mobile offloading systems face the mirror-image problem. They run a weak local model, find it unsure, and then upload the raw image to a server, so the local work is wasted. We propose to connect the two. When an on-chip exit fails, the device compresses the activation tensor it has already computed to a bit budget picked for that task, and streams it to an edge server that finishes the network. The bit budget is set by a lightweight queue-and-channel controller.
The central question is whether a tensor that is still sitting in on-chip memory, requantized to one or two bits per value and allocated unevenly across channels, can beat simpler options. The competitors are sending a compressed image, sending features at a fixed precision, and staying local, each measured on deadline hit rate and accuracy under a fluctuating wireless link. We treat FPGA acceleration as the execution substrate, mobile resource limits as the operating regime, and model cost as the quantity the controller trades against delay.
Core Claim
On an FPGA device running an early-exit network in the "parallel" arrangement of Realization of Early-Exit Dynamic Neural Networks on Reconfigurable Hardware, the tensor at a failed exit can be sent off-chip at a variable average bit budget $b$ (bits per activation). It is quantized by a small hardware unit that splits $b$ across channels using the analytical per-channel bit allocation of Post training 4-bit quantization of convolutional networks for rapid-deployment, and some channels may receive zero bits. The edge-side remainder of the network is fine-tuned to tolerate the full range of budgets. A per-task controller chooses $b$, and whether to offload at all, from the exit's confidence, the current link rate and the device's backlog.
We claim this combination achieves higher accuracy at a fixed deadline-hit rate than three alternatives: raw-image offload, fixed-precision feature offload, and learned-autoencoder feature offload. The margin should be largest when bandwidth swings by an order of magnitude. The new element is not early exits, feature offloading or quantization separately. It is a rate-controlled feature codec driven by exit confidence and queue state, sitting at the exit boundary of an FPGA accelerator.
How This Differs From Existing Work
Several papers motivate the gap. The FPGA early-exit work above shows that running the exit head concurrently with the next backbone layer keeps activations on-chip and yields 1.5x to 3x speedups over static FPGA designs. It also reports at most 1.3% loss from 8-bit fixed point and 4x better energy efficiency than a Jetson Xavier NX. A failed exit there only ever means continuing locally, with no link involved.
Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approach names the "fail-upward" waste of computing locally and then offloading from scratch. It avoids that waste by routing BERT inputs upfront, in floating point, with no hardware accelerator. Our answer is different: we make the local work reusable rather than skipping it.
The Case for FPGA-Based Edge Computing puts the FPGA at the edge and offloads whole tasks from the phone, reporting up to 29.5% device-energy savings. We put the FPGA on the device and split the network across the link.
A second group supplies the closest systems, and each has a component we replace. Early-Exit Meets Model-Distributed Inference at Edge Networks sends exit-point features between Jetson Nano boards through a fixed autoencoder, costing up to 2.2% accuracy, and tunes thresholds with heuristics. We use a rate-adjustable quantizer with no learned encoder, on FPGA logic. Joint DNN partitioning and task offloading in mobile edge computing via deep reinforcement learning learns cut points with PPO but ships unquantized tensors and models compute as generic CPU cycles. Adaptive Edge Offloading for Image Classification Under Rate Limit gates raw-image offload on local entropy with a DQN. Dynamic Compression Ratio Selection for Edge Inference Systems with Hard Deadlines adapts input resolution to deadlines and adds server-requested retransmission. Its idea of trading fidelity for timeliness is the one we apply in feature space instead of pixel space.
The controller borrows its method from the Lyapunov drift-plus-penalty policies of Stochastic Joint Radio and Computational Resource Management for Multi-User Mobile-Edge Computing Systems and Power-Delay Tradeoff in Multi-User Mobile-Edge Computing Systems. Those papers control transmit power and CPU frequency. We control bit budget and exit decision instead.
We deliberately do not vary the precision of the FPGA's arithmetic, as a bit-serial overlay like BISMO: A Scalable Bit-Serial Matrix Multiplication Overlay for Reconfigurable Computing would allow. Only the transmitted bits interact with the channel. Tying compute precision to link quality would add hardware complexity and accuracy loss for no transmission benefit. Compute stays at the 8-bit systolic array that is already proven.
Research Program
Network and training. We train early-exit versions of ResNet32 and VGG19 for CIFAR-10 and CIFAR-100, and ResNet-50 for ImageNet, placing one or two exits. The edge-side remainder of each network is fine-tuned with quantization noise sampled over budgets from 0.5 to 8 bits. Allocations are computed with Banner-style analytic per-channel allocation from calibration statistics. Because activation statistics shift in deployment, a known weakness raised in that paper, the clipping thresholds are recomputed per tensor from running channel ranges in hardware rather than fixed offline.
Hardware. On a Xilinx ZCU106, which stands in for a mobile-class FPGA device, we extend the parallel early-exit design with an exit-boundary codec. The codec reads the in-flight tensor from on-chip buffers, computes per-channel ranges, applies the allocated clipping and requantization, packs bits and hands them to the network interface through DMA. We measure the codec's added latency, logic usage and power separately from the rest of the design.
Controller. For task $t$ with exit confidence $c_t$, link rate estimate $r_t$ and device backlog $Q_t$, the controller picks an action $a_t$. The action is either "continue locally" or "offload at budget $b$", and the controller chooses it to minimize
$$V\,\big[\Delta\mathrm{acc}(a_t \mid c_t) + \lambda E(a_t)\big] + Q_t\, D(a_t, r_t).$$
Here $\Delta\mathrm{acc}$ is the expected accuracy loss, looked up from an offline profile indexed by confidence bin and budget. $E$ is device energy, $D$ is predicted completion delay, $\lambda$ weights energy against accuracy, and $V$ sets the accuracy-versus-backlog tradeoff that yields the $[O(1/V), O(V)]$ behavior of the cited Lyapunov work.
Drift-plus-penalty bounds only average backlog. Hard deadlines are therefore enforced separately, by discarding any action whose predicted $D$ exceeds the remaining slack. We state openly that this filter is only as good as the rate estimate. The policy needs no training, and its decision is a scan over roughly twenty actions, far cheaper than the 0.25 ms DQN decision reported in the rate-limit offloading paper.
Environment. A real WiFi link between the board and a GPU edge server is shaped by replaying synthetic Markov-modulated bandwidth and arrival traces. We use synthetic traces following the rate-limit offloading paper, because the research file lists no public channel-trace benchmark. Deadlines are swept across a range, and packet loss is injected up to 10% to match the compression-ratio paper's stress test.
Baselines
The naive references are two. The first is local-only execution of the full early-exit network on the FPGA, exactly the parallel design of Realization of Early-Exit Dynamic Neural Networks on Reconfigurable Hardware. The second is always offloading the JPEG-compressed input to the edge server's full model. Both run on the same board, server, network weights and traces.
The strongest existing method for deadline-bound adaptive offloading is Dynamic Compression Ratio Selection for Edge Inference Systems with Hard Deadlines. It reports a 97.5% timely-inference success rate on MNIST, 4.5% above the best fixed resolution at high arrival rates, and a 5% gain on CIFAR-10 from information augmentation. We reimplement its resolution-selection policy and retransmission on our testbed, using the same server model, trained on the same set of resolutions it chooses among. Beating it means higher accuracy among on-time tasks at an equal or higher deadline-hit rate, across at least the high-arrival, low-bandwidth half of the sweep.
Two further baselines test the feature path directly. One is a fixed-bit feature offload at the same exit (8, 4 and 2 bits), with the same edge-side fine-tuning, which gives the "adaptivity is unnecessary" reference. The other is the autoencoder codec of Early-Exit Meets Model-Distributed Inference at Edge Networks, trained at three matched bitrates. For that one, the file reports only its up-to-2.2% accuracy drop and no deadline metric. The entropy-gated DQN of Adaptive Edge Offloading for Image Classification Under Rate Limit is run with our exit head as its confidence source and raw JPEG as its payload, with the same training trace budget we allow any learned controller.
Ablations isolate the key ingredients. The first removes per-channel allocation, using a uniform bit budget. The second removes budget-robust fine-tuning of the edge-side network. The third lets the controller see the channel but not the exit confidence. The fourth moves the codec from FPGA logic to the board's ARM cores.
Why not just…?
Why not just send the compressed image when the exit fails? This is the most serious threat. On CIFAR, the tensor after an early block is dozens of times larger in raw values than a 32×32 image, so feature offload wins only if the budget falls near or below one bit per value, or if the exit sits deep enough that the tensor has shrunk. The decisive measurement is the rate–accuracy curve of exit features under our codec, compared against JPEG input at matched bytes, for every exit position. If JPEG dominates at all positions on CIFAR, we will say so and restrict the claim to ImageNet-scale inputs and deeper exits. There, the remote side also avoids recomputing the early layers that the device has already paid for.
Why not just fix the feature precision at 4 or 2 bits? If the channel rarely swings, a fixed budget matched to the median rate may be nearly as good, and cheaper to reason about. Banner et al.'s ResNet101 result, rising from 62.6% to 75.0% at 4W4A against a 77.2% FP32 reference, shows that careful quantization leaves a gap, so no single budget is free. The fixed-bit baselines across bandwidth-variance levels will show where adaptivity starts to pay. We expect little gain at low variance.
Why not just learn the policy with PPO or DQN? Learned controllers such as the PPO agent in Joint DNN partitioning and task offloading in mobile edge computing via deep reinforcement learning, which converges in 34 epochs versus 110 for DDQN, can match a hand-built objective on the traces they were trained on. We test both controllers on channel statistics shifted away from training. The claim for the Lyapunov rule is robustness without training, not higher peak accuracy.
Why not just use a learned autoencoder, or requantize in software? A learned codec may compress better per byte, and it is included at matched rates. However, it is fixed-rate and adds its own compute. The ARM-core ablation tests whether FPGA logic matters at all. If software requantization adds negligible latency, the contribution reduces to the codec and controller, and we will report it that way.
Proposed Experiments
The first decisive check is the JPEG-versus-feature rate–accuracy crossover per exit on CIFAR-10, CIFAR-100 and ImageNet. If no exit wins on any dataset, the proposal stops there. The second is the main deadline sweep: deadline-hit rate, accuracy on on-time tasks, and device energy per task, reported against the resolution-selection baseline and fixed-bit offload under low-, medium- and high-variance channel traces. The third is hardware cost: codec latency, logic and memory usage, and whether the parallel design's DRAM-free property survives the added unit. The fourth tests confidence-conditioned budgeting: whether low-confidence tasks benefit from more bits, which would show that the exit head carries information useful for rate control beyond the stop/continue decision.
References
- Realization of Early-Exit Dynamic Neural Networks on Reconfigurable Hardware
- Post training 4-bit quantization of convolutional networks for rapid-deployment
- Distributed Inference on Mobile Edge and Cloud: An Early Exit Based Clustering Approach
- The Case for FPGA-Based Edge Computing
- Early-Exit Meets Model-Distributed Inference at Edge Networks
- Joint DNN partitioning and task offloading in mobile edge computing via deep reinforcement learning
- Adaptive Edge Offloading for Image Classification Under Rate Limit
- Dynamic Compression Ratio Selection for Edge Inference Systems With Hard Deadlines
- Stochastic Joint Radio and Computational Resource Management for Multi-User Mobile-Edge Computing Systems
- Power-Delay Tradeoff in Multi-User Mobile-Edge Computing Systems
- BISMO: A Scalable Bit-Serial Matrix Multiplication Overlay for Reconfigurable Computing