Efficient Edge Inference by Selective Query

Anil Kag, Igor Fedorov, Aditya Gangrade, Paul Whatmough, Venkatesh SaligramaICLR 2023 · OpenReview

Deep Neural Networks (DNNs) have achieved incredible accuracy across a variety of predictive tasks, but deploying them on resource-constrained "edge" devices—like smartwatches, home assistants, or micro-controllers—remains a significant challenge. The standard industry workaround is to send data to the cloud for processing. However, uploading high-dimensional data like images or audio introduces massive communication bottlenecks, leading to high latency and severe battery drain.

While existing "hybrid" solutions attempt to split the workload between the edge and the cloud, they often rely on simple heuristics. For instance, many systems simply send data to the cloud if the edge model is "uncertain" (entropy thresholding). These methods are myopic: they fail to consider whether the cloud model can actually fix the error, and they do not systematically train the edge model, the routing mechanism, and the cloud model to work together as a cohesive unit.

This paper introduces a novel end-to-end hybrid learning framework that optimizes inference by selectively querying the cloud only for "hard" examples that the on-device model cannot handle, provided the cloud can actually solve them.

A Three-Part Hybrid Architecture

The authors propose a framework consisting of three distinct components:

  1. Base Model ($b$): A small, computationally "cheap" model running locally on the edge device.
  2. Global Model ($g$): A large, "expensive" state-of-the-art model running in the cloud.
  3. Router ($r$): A tiny, lightweight model on the edge that decides whether to output the local prediction or transmit the data to query the cloud.
The hybrid framework routes inputs through a local base model and a router, selectively querying the cloud only when necessary.

The Secret Sauce: Proxy Supervision

The core innovation of this paper lies in how the router is trained. Instead of relying on the base model's confidence scores, the authors introduce Proxy Supervision.

They define an "oracle" during training that knows exactly when the cloud should be queried: specifically, when the edge model is incorrect but the cloud model is correct. By training the router to mimic this oracle, the system learns to avoid wasting time, bandwidth, and energy on queries that neither model can solve, or that the edge model has already solved correctly.

The framework utilizes Alternating Optimization to refine these components and a Joint Neural Architecture Search (NAS) to find the optimal pair of edge and cloud architectures for a specific latency budget.

Key Insights and Results

The researchers evaluated their method across diverse tasks, including ImageNet (large-scale image classification), CIFAR-100, and IMDb (text-based sentiment analysis).

  • Coverage as a Key Metric: The authors define "coverage" as the fraction of queries handled entirely on-device. They demonstrate that coverage has a direct, one-to-one correspondence with average latency, making it a powerful proxy for system performance during training.
  • Beyond Simple Uncertainty: By asking "Will the cloud model actually help?" rather than just "Is the edge model confused?", the framework maximizes the "accuracy-per-millisecond" trade-off.
  • Hardware Agnostic: The framework adapts its architecture search to the specific resource constraints of the device, whether it is a tiny micro-controller (MCU) or a powerful smartphone.
The proposed hybrid method Pareto-dominates existing baselines, achieving higher accuracy for the same latency or lower latency for the same accuracy.

The performance gains are substantial:

  • Micro-controller (MCU) Performance: On an ultra-low-power STM32F746 MCU, the hybrid model achieved a 25% reduction in latency and energy compared to cloud-only processing, while maintaining the same high accuracy.
  • Mobile Device Gains: For mobile-class processors using MobileNetV3, the system achieved up to a 70% latency reduction while reaching state-of-the-art accuracy levels.
  • Pareto Dominance: Across tests, the proposed hybrid method consistently "Pareto dominated" existing baselines, proving its efficiency across different latency regimes.

Limitations and Future Work

While highly effective, the framework has a few limitations. The current model assumes the edge device can observe or estimate dynamic network latency to adjust its routing threshold, which can be challenging in highly variable environments. Additionally, the Joint NAS relies on pre-trained super-nets; extending this to entirely new, untrained architectures could be computationally expensive. Finally, the system still fundamentally relies on a cloud connection. Future work may explore "multi-tier" hierarchies (e.g., Edge $\rightarrow$ Gateway $\rightarrow$ Cloud) to provide even more granular performance trade-offs.

Summary updated

Abstract

Edge devices provide inference on predictive tasks to many end-users. However, deploying deep neural networks that achieve state-of-the-art accuracy on these devices is infeasible due to edge resource constraints. Nevertheless, cloud-only processing, the de-facto standard, is also problematic, since uploading large amounts of data imposes severe communication bottlenecks. We propose a novel end-to-end hybrid learning framework that allows the edge to selectively query only those hard examples that the cloud can classify correctly. Our framework optimizes over neural architectures and trains edge predictors and routing models so that the overall accuracy remains high while minimizing the overall latency. Training a hybrid learner is difficult since we lack annotations of hard edge-examples. We introduce a novel proxy supervision in this context and show that our method adapts seamlessly and near optimally across different latency regimes. On the ImageNet dataset, our proposed method deployed on a micro-controller unit exhibits $25\%$ reduction in latency compared to cloud-only processing while suffering no excess loss.
BibTeX
@inproceedings{kag2023efficient,
  author = {Anil Kag and Igor Fedorov and Aditya Gangrade and Paul Whatmough and Venkatesh Saligrama},
  title = {{Efficient Edge Inference by Selective Query}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=jpR98ZdIm2q},
}
Figure