Multi-Model Running Latency Optimization in an Edge Computing Paradigm

Peisong Li, Xinheng Wang, Kaizhu Huang, Yi Huang, Shancang Li, Muddesar Iqbal

TL;DR

This paper tackles the challenge of high latency in edge AI systems that run multiple models concurrently, particularly for applications like autonomous driving. They propose a novel scheduling strategy that dynamically assigns tasks to different applications based on their current workload, significantly reducing overall system latency without compromising performance.

What Problem Does This Paper Address?

Modern applications, especially in areas like autonomous driving, rely heavily on AI models running at the network edge to process sensor data in real-time. These systems often involve multiple AI models working in parallel to achieve a common goal (e.g., detecting vehicles, pedestrians, and traffic lights). However, running these models concurrently on resource-constrained edge devices can lead to significant overall latency. This latency is a critical bottleneck, as it directly impacts the system's ability to react quickly and safely. The problem is exacerbated by the heterogeneity of edge hardware (CPUs, GPUs, FPGAs) and the need for models to be compatible across these platforms. Existing solutions often focus on optimizing individual model performance or accuracy, but not on the collaborative latency reduction needed for multi-model inference on a single edge device.

Methods

The core of this paper is a novel multi-model task scheduling strategy designed to minimize overall running latency. The proposed system consists of:

  • Runtime Optimizer: This component is responsible for receiving requests and executing the scheduling strategy.
  • Containerization: Each application (App) is packaged as a container, which helps manage hardware heterogeneity and simplifies deployment.
  • ONNX Runtime: Models are converted to the ONNX format for efficient, cross-platform inference on edge devices.

The paper introduces two main scheduling strategies:

  1. Optimal Solution Selection Method (OSSM): This exhaustive approach evaluates all possible task-to-app assignments in each round to find the one that yields the absolute minimum latency. While optimal, it has a high time complexity (O(n!)).
  2. Simplest Allocation Method (SAM): This more efficient strategy ranks sensor inputs by their processing latency and apps by their current accumulated running latency. It then assigns the highest latency task to the app with the lowest current latency, and so on. This approach significantly reduces complexity (O(n log n)) while still achieving substantial latency reduction.

The system architecture involves sensors feeding data to the edge device, where the Runtime Optimizer schedules tasks to different Apps. Each App then invokes the appropriate ONNX-formatted AI model for inference. Finally, the results from all Apps are fused for a collaborative decision.

Datasets

Not discussed in the paper. The experiments were conducted using pre-trained deep learning models (MobileNet, ShuffleNet, SqueezeNet) deployed on an NVIDIA Jetson Nano.

Key Insights and Ideas

  • Collaborative Latency is Key: For multi-model systems, the overall latency is dictated by the slowest component, not the sum of individual latencies. Optimizing individual models isn't enough; the scheduling of tasks across models is crucial.
  • Dynamic Task Allocation Reduces Idle Time: By dynamically assigning tasks to applications based on their current workload, the system avoids situations where some applications are idle while others are overloaded, leading to more efficient resource utilization and reduced overall latency.
  • ONNX and Containerization Enable Heterogeneity Management: Converting models to ONNX and deploying apps in containers are essential for handling diverse hardware and simplifying the deployment of multi-model systems on edge devices.
  • Trade-off Between Optimality and Efficiency: The paper highlights the classic trade-off between finding the absolute optimal solution (OSSM) and achieving practical efficiency with a near-optimal solution (SAM).

Key Results

  • Significant Latency Reduction: The proposed scheduling strategies (Strategy 1 and Strategy 2) reduced overall running time by 16.7% and 14.3%, respectively, compared to a system without scheduling.
  • Overall Running Time: Without scheduling, the system took 267,848 ms for 1000 inference tasks. With Strategy 1, it was 229,617 ms, and with Strategy 2, it was 223,255 ms.
  • Balanced Resource Utilization: CPU usage was distributed relatively evenly among the three applications, indicating efficient use of the edge device's resources.
  • Slight Increase in Individual Inference Time: While overall system latency decreased, the inference time for a single task slightly increased due to the overhead of the scheduling mechanism. However, this increase was outweighed by the reduction in overall system latency.

Limitations and Future Work

  • Limitations:
    • The paper notes that the inference time of a single task is slightly increased due to the scheduling overhead.
  • The memory footprint increased slightly after adopting the scheduling strategy, though runtime latency was significantly reduced.
  • Future Work:
    • The authors plan to use Reinforcement Learning (RL) to dynamically optimize multi-model task allocation strategies, enabling autonomous learning and adaptation to application states and task characteristics.

Summary updated

BibTeX
@inproceedings{li2022multi,
  author = {Peisong Li and Xinheng Wang and Kaizhu Huang and Yi Huang and Shancang Li and Muddesar Iqbal},
  title = {{Multi-Model Running Latency Optimization in an Edge Computing Paradigm}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=g5-iLEv6O3},
}
Figure