Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU

Zhihe Zhao, Neiwen Ling, Nan Guan, Guoliang Xing

Abstract

Edge computing environments increasingly require the simultaneous execution of multiple Deep Neural Networks (DNNs). However, resource-constrained edge GPUs often struggle with "resource contention," where concurrent tasks compete for internal hardware, leading to unpredictable latency for critical real-time operations. We present Miriam, a novel framework that transforms standard, rigid GPU kernels into "elastic" versions. By dynamically resizing these kernels at runtime, Miriam prioritizes critical tasks while utilizing idle resources for background workloads. Our evaluation demonstrates that Miriam can increase system throughput by up to 92% while maintaining a latency overhead of less than 10% for critical tasks.

Introduction

In autonomous systems, such as self-driving cars or augmented reality headsets, the GPU must manage a diverse set of workloads. These tasks are typically categorized by their importance: critical tasks (e.g., obstacle detection) require strict real-time guarantees, while normal tasks (e.g., pose estimation or fatigue analysis) are less time-sensitive but still essential for overall system functionality.

On server-grade GPUs, technologies like NVIDIA’s Multi-Process Service (MPS) or Multi-Instance GPU (MIG) provide hardware-level isolation to manage these workloads. However, edge GPUs—such as the NVIDIA Jetson series—lack these features. This creates a dilemma: running tasks sequentially ensures low latency for the leader but results in poor GPU utilization, while running them concurrently leads to "resource tug-of-war" over Streaming Multiprocessors (SMs), registers, and memory bandwidth, causing critical tasks to miss their deadlines.

The Miriam Framework

Miriam addresses the contention problem through a three-tiered approach that introduces flexibility into the traditionally rigid execution model of GPU kernels.

1. Elastic Kernel Generation

Standard GPU kernels are typically compiled with fixed grid and block sizes, making them difficult to resize without breaking the underlying mathematical operations. The authors found that only 7.4% of standard kernels are natively "elastic." To overcome this, Miriam employs a source-to-source transformer. This tool automatically injects code into the kernels to handle memory offsets and thread identification. This ensures that the computation remains consistent regardless of how the kernel is partitioned or "sharded" across the GPU.

2. Design Space Shrinking

Partitioning a kernel creates millions of possible configurations. To avoid the overhead of searching for the best configuration at runtime, Miriam uses an offline profiling stage. It employs two custom metrics:

  • WIScore (Workload Imbalance Score): Measures how evenly work is distributed across the GPU.
  • OScore (Overhead Score): Estimates the performance penalty of sharding. By using these metrics, Miriam prunes approximately 80% of the least efficient configurations, leaving only high-performance candidates for the runtime scheduler.

3. Dynamic Runtime Coordinator

At runtime, Miriam manages kernel shards using a shaded binary tree structure. When a critical task arrives, the coordinator grants it the necessary resources immediately. It then "pads" the remaining GPU capacity with small shards of normal tasks. This "padding" philosophy treats critical tasks as first-class citizens while using normal tasks as filler to maximize throughput without interfering with the critical path.

Evaluation and Results

The framework was evaluated using the Mixed-critical DNN Task Benchmarks (MDTB), which includes models like ResNet, AlexNet, and SqueezeNet, as well as real-world traces from the LGSVL autonomous driving simulator.

Throughput and Latency

Miriam was tested on both the NVIDIA RTX 2060 and the Jetson AGX Xavier. The results show a significant performance leap:

  • Throughput: System-wide throughput increased by up to 92% compared to sequential execution.
  • Latency: Critical tasks maintained high predictability, with less than 10% latency overhead in most scenarios. In contrast, standard multi-streaming approaches often saw latency spikes exceeding 100% under heavy contention.
  • Utilization: By adjusting both block sizes (to manage intra-SM contention) and grid sizes (to manage inter-SM contention), Miriam achieved significantly higher SM occupancy than traditional baselines.

Conclusion and Future Work

Miriam demonstrates that kernel elasticity is a viable path toward balancing real-time requirements with high throughput on edge devices. By transforming the execution paradigm from rigid blocks to dynamic shards, it effectively mitigates the resource contention inherent in multi-DNN workloads.

Future iterations of the framework will look to address remaining overheads (approximately 15μs) caused by contention in the texture cache and L2 memory. Additionally, the team aims to scale the scheduling policies to support more than two simultaneous models and further automate the offline profiling process to support rapid deployment of custom DNN architectures.

Summary updated

BibTeX
@inproceedings{zhao2023miriam,
  author = {Zhihe Zhao and Neiwen Ling and Nan Guan and Guoliang Xing},
  title = {{Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU}},
  booktitle = {SenSys},
  year = {2023},
  url = {https://openreview.net/forum?id=zRJHLPt0ll},
}
Figure