Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU

Zhihe Zhao, Neiwen Ling, Nan Guan, Guoliang Xing

Abstract

Edge computing environments increasingly require the simultaneous execution of multiple Deep Neural Networks (DNNs). However, resource-constrained edge GPUs often struggle with "resource contention," where concurrent tasks compete for internal hardware, leading to unpredictable latency for critical real-time operations. We present Miriam, a novel framework that transforms standard, rigid GPU kernels into "elastic" versions. By dynamically resizing these kernels at runtime, Miriam prioritizes critical tasks while utilizing idle resources for background workloads. Our evaluation demonstrates that Miriam can increase system throughput by up to 92% while maintaining a latency overhead of less than 10% for critical tasks.

Introduction

In autonomous systems, such as self-driving cars or augmented reality headsets, the GPU must manage a diverse set of workloads. These tasks are typically categorized by their importance: critical tasks (e.g., obstacle detection) require strict real-time guarantees, while normal tasks (e.g., pose estimation or fatigue analysis) are less time-sensitive but still essential for overall system functionality.

On server-grade GPUs, technologies like NVIDIA’s Multi-Process Service (MPS) or Multi-Instance GPU (MIG) provide hardware-level isolation to manage these workloads. However, edge GPUs—such as the NVIDIA Jetson series—lack these features. This creates a dilemma: running tasks sequentially ensures low latency for the leader but results in poor GPU utilization, while running them concurrently leads to "resource tug-of-war" over Streaming Multiprocessors (SMs), registers, and memory bandwidth, causing critical tasks to miss their deadlines.

The Miriam Framework

Miriam addresses the contention problem through a three-tiered approach that introduces flexibility into the traditionally rigid execution model of GPU kernels.

1. Elastic Kernel Generation

Standard GPU kernels are typically compiled with fixed grid and block sizes, making them difficult to resize without breaking the underlying mathematical operations. The authors found that only 7.4% of standard kernels are natively "elastic." To overcome this, Miriam employs a source-to-source transformer. This tool automatically injects code into the kernels to handle memory offsets and thread identification. This ensures that the computation remains consistent regardless of how the kernel is partitioned or "sharded" across the GPU.

2. Design Space Shrinking

Partitioning a kernel creates millions of possible configurations. To avoid the overhead of searching for the best configuration at runtime, Miriam uses an offline profiling stage. It employs two custom metrics:

  • WIScore (Workload Imbalance Score): Measures how evenly work is distributed across the GPU.
  • OScore (Overhead Score): Estimates the performance penalty of sharding. By using these metrics, Miriam prunes approximately 80% of the least efficient configurations, leaving only high-performance candidates for the runtime scheduler.

3. Dynamic Runtime Coordinator

At runtime, Miriam manages kernel shards using a shaded binary tree structure. When a critical task arrives, the coordinator grants it the necessary resources immediately. It then "pads" the remaining GPU capacity with small shards of normal tasks. This "padding" philosophy treats critical tasks as first-class citizens while using normal tasks as filler to maximize throughput without interfering with the critical path.

Evaluation and Results

The framework was evaluated using the Mixed-critical DNN Task Benchmarks (MDTB), which includes models like ResNet, AlexNet, and SqueezeNet, as well as real-world traces from the LGSVL autonomous driving simulator.

Throughput and Latency

Miriam was tested on both the NVIDIA RTX 2060 and the Jetson AGX Xavier. The results show a significant performance leap:

  • Throughput: System-wide throughput increased by up to 92% compared to sequential execution.
  • Latency: Critical tasks maintained high predictability, with less than 10% latency overhead in most scenarios. In contrast, standard multi-streaming approaches often saw latency spikes exceeding 100% under heavy contention.
  • Utilization: By adjusting both block sizes (to manage intra-SM contention) and grid sizes (to manage inter-SM contention), Miriam achieved significantly higher SM occupancy than traditional baselines.

Conclusion and Future Work

Miriam demonstrates that kernel elasticity is a viable path toward balancing real-time requirements with high throughput on edge devices. By transforming the execution paradigm from rigid blocks to dynamic shards, it effectively mitigates the resource contention inherent in multi-DNN workloads.

Future iterations of the framework will look to address remaining overheads (approximately 15μs) caused by contention in the texture cache and L2 memory. Additionally, the team aims to scale the scheduling policies to support more than two simultaneous models and further automate the offline profiling process to support rapid deployment of custom DNN architectures.

Summary updated

Abstract

Many applications such as autonomous driving and augmented reality, require the concurrent running of multiple deep neural networks (DNN) that poses different levels of real-time performance requirements. However, coordinating multiple DNN tasks with varying levels of criticality on edge GPUs remains an area of limited study. Unlike server-level GPUs, edge GPUs are resource-limited and lack hardware-level resource management mechanisms for avoiding resource contention. Therefore, we propose Miriam, a contention-aware task coordination framework for multi-DNN inference on edge GPU. Miriam consolidates two main components, an elastic-kernel generator, and a runtime dynamic kernel coordinator, to support mixed critical DNN inference. To evaluate Miriam, we build a new DNN inference benchmark based on CUDA with diverse representative DNN workloads. Experiments on two edge GPU platforms show that Miriam can increase system throughput by 92% while only incurring less than 10\% latency overhead for critical tasks, compared to state of art baselines.
BibTeX
@inproceedings{zhao2023miriam,
  author = {Zhihe Zhao and Neiwen Ling and Nan Guan and Guoliang Xing},
  title = {{Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU}},
  booktitle = {SenSys},
  year = {2023},
  url = {https://openreview.net/forum?id=zRJHLPt0ll},
}
Figure