Corun: Concurrent Inference and Continuous Training at the Edge for Cost-Efficient AI-Based Mobile Image Sensing

Yu Liu, Anurag Andhare, Kyoung‐Don KangSensors 2024 · DOI

TL;DR

This paper introduces "Corun," a novel framework for edge computing that allows a single GPU to simultaneously handle multiple AI image inference requests and continuous model retraining. This approach significantly boosts inference throughput and maintains accuracy without requiring expensive, dedicated hardware for each task, making AI-powered mobile image sensing more cost-efficient.

What Problem Does This Paper Address?

Mobile devices, while increasingly capable, often lack the computational power and battery life for complex AI tasks like image analysis. Offloading these tasks to edge servers offers a solution, but it faces challenges:

  • Resource Constraints: Edge servers, while more powerful than mobile devices, still have limited resources. Running multiple AI inference tasks (e.g., image classification, object detection) concurrently on a single GPU can lead to increased latency and out-of-memory (OOM) errors.
  • Model Accuracy Degradation: AI models can experience accuracy drops over time due to "data drift" – changes in the real-world data distribution compared to the training data. Continuous retraining is needed to maintain accuracy, but this further strains GPU resources.
  • Cost Inefficiency: Traditional approaches often dedicate a separate GPU to each inference task or to the retraining process, leading to high hardware costs and underutilization of resources.

The paper aims to bridge this gap by enabling efficient co-execution of both inference and retraining on a single, cost-effective edge GPU.

Methods

Corun's core innovation lies in its intelligent scheduling and resource management strategy:

  • Offline Profiling (Algorithm 1): Before deployment, Corun performs an offline profiling step. It systematically determines the maximum number of concurrent inference tasks (K) that can run alongside a continuous retraining job on a specific GPU without causing OOM errors or a superlinear increase in latency. This process is crucial for understanding the GPU's capacity and limitations.
  • Spatial GPU Multiplexing: Corun utilizes NVIDIA's Multi-Process Service (MPS) to create multiple CUDA streams, effectively dividing the GPU's resources among the concurrent inference tasks and the retraining job. This allows them to run in parallel rather than sequentially.
  • FCFS Scheduling with Multiplexed Dispatch: At runtime, Corun employs a First-Come, First-Served (FCFS) scheduling approach. When inference requests arrive, Corun dispatches up to the pre-determined maximum number (K) of requests to run concurrently with the ongoing training job.
  • Batch Size Optimization: The paper discusses the trade-offs of batch sizes for both inference (batch size of 1 for low latency) and training (relatively small batch sizes to balance concurrency and training reliability).

A key aspect is that Corun does not require modifications to the underlying GPU hardware, drivers, or DL frameworks, making it broadly applicable.

Datasets

  • ImageNet [58]: Used for pre-training CNN models for inference tasks.
  • NH-HAZE [59]: Used for evaluating the performance of image dehazing Transformer models.
  • Mini-ImageNet [60]: Used for continuous retraining of CNN models. This dataset is smaller than ImageNet, simulating the scenario of periodic updates with fresh sample images on an edge server. The paper also stresses-tests Corun by using the full 50,000 training images per epoch.

Key Insights and Ideas

  • GPU Underutilization: The pilot study revealed that typical solo training and inference jobs on a commodity GPU (NVIDIA RTX 3080 Ti) utilize only a fraction of its capacity, indicating significant room for concurrent execution.
  • Memory as the Bottleneck: GPU memory is identified as the primary bottleneck for co-executing multiple tasks. Corun's offline profiling is designed to avoid OOM errors by finding a feasible concurrency level.
  • Sublinear Latency Increase: A critical finding is that while inference throughput scales linearly with the number of concurrent tasks, the increase in inference latency and training epoch time is sublinear. This means you get much more work done for a relatively small performance penalty.
  • Cost-Effectiveness: By enabling multiple tasks on a single GPU, Corun significantly reduces the number of GPUs required, leading to substantial cost savings for deploying AI-powered mobile image sensing solutions.

Key Results

  • Inference Throughput: Corun can scale inference throughput (QPS) linearly, achieving up to 4.69x improvement for CNN models and up to 3.7x for Transformer models when running 4 concurrent inference tasks alongside training.
  • Inference Latency: The normalized average inference latency increases sublinearly, by at most 1.13x for CNNs and 1.28x for Transformers, even with increased concurrency. The 95-P tail latency also shows manageable increases.
  • Training Epoch Time: The epoch length for training also increases sublinearly, by up to 1.82x for CNNs when running with 4 concurrent inference tasks. For Transformer models, the impact on training epoch time was found to be insignificant, potentially due to the use of a more powerful GPU (RTX 4090).
  • OOM Error Avoidance: Corun successfully avoids OOM errors through its offline profiling and scheduling mechanism.

Limitations and Future Work

  • UVM Disabled: For low latency, Unified Virtual Memory (UVM) was disabled. Future work could explore its potential to allow more tasks by swapping between GPU and host memory, provided latency impacts can be managed.
  • Adaptive Retraining: The current work assumes continuous retraining. Future research could focus on adaptive retraining triggered by detected data drift, allowing for more concurrent inferences during periods of low drift.
  • On-Device Inference & Offloading: The paper assumes all queries are offloaded. Exploring hybrid approaches where lightweight inference happens on-device and heavier tasks are offloaded to the edge could be beneficial.
  • Autoscaling Integration: Corun could be combined with existing autoscaling solutions to dynamically allocate more GPUs when needed, especially for handling sudden spikes in inference requests.
  • Vision Transformers (ViTs): ViTs are computationally more demanding than CNNs, leading to higher latency. Future work could explore model compression techniques for ViTs to make them more suitable for edge deployment alongside Corun.
  • Fault Isolation: The current approach relies on MPS, which offers less robust fault isolation compared to hardware-level partitioning (like NVIDIA MIG). Future work could investigate virtualization techniques or cluster-level scheduling for better fault tolerance.

Summary updated

Abstract

Intelligent mobile image sensing powered by deep learning analyzes images captured by cameras from mobile devices, such as smartphones or smartwatches. It supports numerous mobile applications, such as image classification, face recognition, and camera scene detection. Unfortunately, mobile devices often lack the resources necessary for deep learning, leading to increased inference latency and rapid battery consumption. Moreover, the inference accuracy may decline over time due to potential data drift. To address these issues, we introduce a new cost-efficient framework, called Corun, designed to simultaneously handle multiple inference queries and continual model retraining/fine-tuning of a pre-trained model on a single commodity GPU in an edge server to significantly improve the inference throughput, upholding the inference accuracy. The scheduling method of Corun undertakes offline profiling to find the maximum number of concurrent inferences that can be executed along with a retraining job on a single GPU without incurring an out-of-memory error or significantly increasing the latency. Our evaluation verifies the cost-effectiveness of Corun. The inference throughput provided by Corun scales with the number of concurrent inference queries. However, the latency of inference queries and the length of a retraining epoch increase at substantially lower rates. By concurrently processing multiple inference and retraining tasks on one GPU instead of using a separate GPU for each task, Corun could reduce the number of GPUs and cost required to deploy mobile image sensing applications based on deep learning at the edge.
BibTeX
@inproceedings{liu2024corun,
  author = {Yu Liu and Anurag Andhare and Kyoung‐Don Kang},
  title = {{Corun: Concurrent Inference and Continuous Training at the Edge for Cost-Efficient AI-Based Mobile Image Sensing}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24165262},
}
Figure