Autodidactic Neurosurgeon: Collaborative Deep Inference for Mobile Edge Intelligence via Online Learning

Letian Zhang, Lixing Chen, Jie Xu

Abstract

Autodidactic Neurosurgeon (ANS) is an online learning system designed to optimize Deep Neural Network (DNN) partitioning between mobile devices and edge servers in real-time. By employing a specialized contextual bandit algorithm, $\mu$LinUCB, ANS dynamically identifies the optimal "cut point" in a model to minimize end-to-end latency. Unlike traditional methods that rely on static offline profiling, ANS adapts to fluctuating network conditions and server workloads on-the-fly without prior knowledge. Experimental results demonstrate that ANS can reduce inference delay by nearly 30% compared to baseline strategies, converging to optimal performance within just 80 video frames.

Introduction

Deep learning on mobile devices is constrained by a "Goldilocks" problem: local processing is often too slow due to hardware limitations, while full offloading to the cloud can be hindered by low network bandwidth. Collaborative inference—splitting a DNN so the mobile device handles initial layers and the edge server processes the remainder—presents a viable middle ground.

However, the optimal partition point is a moving target. As network speeds fluctuate and server workloads vary, the best split point shifts. Traditional solutions rely on extensive offline profiling, which fails to account for real-world dynamics. There is a critical need for a system that can learn and adapt to these environmental changes autonomously.

End-to-end delay for different DNN partition points under varying uplink conditions (Low, Medium, High).

Methods

The Autodidactic Neurosurgeon (ANS) framework treats the partitioning challenge as a Contextual Multi-Armed Bandit problem. The system architecture consists of a mobile front-end, an edge back-end, and a central Learning Module.

Overview of the Autodidactic Neurosurgeon (ANS) system architecture and learning module.

The $\mu$LinUCB Algorithm

At the heart of ANS is $\mu$LinUCB, a specialized version of the Linear Upper Confidence Bound algorithm. It predicts offloading delays based on contextual features of each potential partition point, such as the number of remaining layers, multiply-accumulate (MAC) operations, and the size of intermediate data. This feature-based approach allows for contextual knowledge transfer; the system can infer the performance of unseen partition points based on their similarity to previously tested ones.

Overcoming the "Local Trap"

A significant challenge in online learning for offloading is the "local trap": if the system decides local processing is best, it stops offloading and loses visibility into network or server improvements. ANS addresses this via a forced sampling mechanism, which periodically tests the server even when local processing appears superior, ensuring the system can detect and exploit improved environmental conditions.

Differentiated Service

ANS further optimizes performance by distinguishing between video frames. Using the Structural Similarity Index (SSIM), it identifies "key frames" that carry more information. The system prioritizes these frames, adopting a more conservative exploration strategy to ensure they experience minimal latency.

Results and Discussion

The system was evaluated using Vgg16, YoLo, and ResNet50 architectures on a testbed featuring an NVIDIA Jetson TX2 and a GPU workstation.

  • Latency and Accuracy: For Vgg16, ANS reduced end-to-end delay by 29.64% compared to static local or full-offloading strategies. The system achieved high prediction accuracy, with errors ranging from 0.39% to 3.12%, significantly outperforming traditional layer-wise summation methods which can have errors exceeding 50% due to the neglect of inter-layer optimizations.
  • Convergence and Efficiency: ANS starts with zero prior knowledge but converges to near-optimal "Oracle" performance within approximately 80 frames. Furthermore, the $\mu$LinUCB algorithm is computationally lightweight, imposing negligible overhead on the mobile device.
  • Adaptability: In dynamic network scenarios, ANS successfully detected network recovery and adjusted its partitioning strategy within 80 frames, whereas standard algorithms without forced sampling remained stuck in inefficient local processing.

Conclusion

ANS provides a robust, self-learning solution for collaborative DNN inference in dynamic edge environments. By moving away from brittle offline profiling towards real-time online learning, it enables mobile devices to consistently achieve low-latency deep inference. Future work will focus on extending ANS to multi-user scenarios with shared server resources and incorporating energy-awareness into the partitioning logic to better serve battery-constrained mobile devices.

Summary updated

Abstract

Recent breakthroughs in deep learning (DL) have led to the emergence of many intelligent mobile applications and services, but in the meanwhile also pose unprecedented computing challenges on resource-constrained mobile devices. This paper builds a collaborative deep inference system between a resource-constrained mobile device and a powerful edge server, aiming at joining the power of both on-device processing and computation offloading. The basic idea of this system is to partition a deep neural network (DNN) into a front-end part running on the mobile device and a back-end part running on the edge server, with the key challenge being how to locate the optimal partition point to minimize the end-to-end inference delay. Unlike existing efforts on DNN partitioning that rely heavily on a dedicated offline profiling stage to search for the optimal partition point, our system has a built-in online learning module, called Autodidactic Neurosurgeon (ANS), to automatically learn the optimal partition point on-the-fly. Therefore, ANS is able to closely follow the changes of the system environment by generating new knowledge for adaptive decision making. The core of ANS is a novel contextual bandit learning algorithm, called $\mu$LinUCB, which not only has provable theoretical learning performance guarantee but also is ultra-lightweight for easy real-world implementation. We implement our system on a video stream object detection testbed to validate the design of ANS and evaluate its performance. The experiments show that ANS significantly outperforms state-of-the-art benchmarks in terms of tracking system changes and reducing the end-to-end inference delay.
BibTeX
@inproceedings{zhang2021autodidactic,
  author = {Letian Zhang and Lixing Chen and Jie Xu},
  title = {{Autodidactic Neurosurgeon: Collaborative Deep Inference for Mobile Edge Intelligence via Online Learning}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442381.3450051},
}
Figure