DynaMIX: Resource Optimization for DNN-Based Real-Time Applications on a Multi-Tasking System

Minkyoung Cho, Kang G. Shin

Introduction

Autonomous vehicles (AVs) rely on a suite of Deep Neural Network (DNN) applications—such as object detection, lane tracking, and sign classification—running simultaneously. These applications share a common "heartbeat": the camera's frame rate. If a camera captures 30 frames per second, every application must finish processing the current frame before the next one arrives.

The challenge is that onboard resources (CPU, GPU, memory) are limited. When the number of active apps changes—for instance, when a car enters a busy intersection and triggers more specialized detection tasks—the system can become overloaded. Standard multi-tasking systems often miss deadlines because they treat DNN models as static "black boxes."

Comparison of task execution timing. Multi-tasking without optimization leads to deadline misses, whereas DynaMIX aims to fit all tasks within the frame-rate window.

The DynaMIX Framework

The authors propose DynaMIX (Dynamic MIXed-precision model construction), a resource optimization framework designed to ensure concurrent deep learning applications meet strict real-time deadlines. By combining mixed-precision model construction with a "lazy loading" mechanism, it maximizes execution accuracy while preventing the "deadline misses" that occur when multiple heavy models compete for limited onboard compute.

The DynaMIX framework architecture, illustrating the transition from offline profiling and model building to online resource allocation and execution.

1. Offline Profiling and Model Building

For each application, the system builds a library of mixed-precision models. It identifies which layers are most sensitive to quantization (using KL-Divergence) and keeps them in high precision (FP32) while quantizing less sensitive layers to INT8. It then generates "performance profiles"—mathematical curves (cubic polynomials) that predict the latency and accuracy of a model based on its memory footprint.

2. Online Resource Allocation

When the workload changes, a main process solves a constrained nonlinear optimization problem. It finds the optimal memory allocation for each app that maximizes total system accuracy while ensuring the sum of all app latencies stays below the frame-rate deadline. By using polynomial regression to create "profiles" offline, the online controller can make complex allocation decisions in milliseconds.

3. Lazy Loading Reconfiguration

To avoid the massive overhead of swapping entire models in and out of memory, DynaMIX uses "lazy loading." It keeps a base INT8 version of the model in memory and only loads the specific high-precision (FP32) layers required for the newly selected configuration at runtime. This allows the system to reconfigure a model's precision on-the-fly without stalling the execution pipeline.

Experimental Evaluation

The framework was evaluated using three representative DNN models and their corresponding standard benchmarks: VGG16 (CIFAR-10), ResNet50 (ImageNet), and YOLO-v3 (MS COCO).

Key results include:

  • Deadline Satisfaction: In scenarios where a standard baseline would miss deadlines by hundreds of milliseconds, DynaMIX successfully reconfigured models to stay within the 250ms–700ms target windows.
  • High Accuracy: Despite aggressive compression, the total accuracy drop across all concurrent apps was kept under 2.2%.
  • Efficiency Gains: For the YOLO-v3 model, the most compressed version reduced latency by 4.3x and memory usage by 1.5x compared to the full-precision baseline.
  • Comparison to State-of-the-Art: Compared to NestDNN (a leading pruning-based approach), DynaMIX achieved significantly higher inference accuracy (92.5% vs 89% for VGG16) because it optimizes for mixed-precision rather than just removing filters.

Conclusion and Future Work

DynaMIX demonstrates that sensitivity-aware mixed-precision is a powerful tool for real-time resource management. Future work will explore relaxing the assumption that all base models must fit in memory simultaneously, incorporating ultra-low precision (INT4/INT2), and implementing "mixed criticality" modes to prioritize safety-critical tasks like emergency braking over less critical functions.

Summary updated

Abstract

As deep neural networks (DNNs) prove their importance and feasibility, more and more DNN-based apps, such as detection and classification of objects, have been developed and deployed on autonomous vehicles (AVs). To meet their growing expectations and requirements, AVs should "optimize" use of their limited onboard computing resources for multiple concurrent in-vehicle apps while satisfying their timing requirements (especially for safety). That is, real-time AV apps should share the limited on-board resources with other concurrent apps without missing their deadlines dictated by the frame rate of a camera that generates and provides input images to the apps. However, most, if not all, of existing DNN solutions focus on enhancing the concurrency of their specific hardware without dynamically optimizing/modifying the DNN apps' resource requirements, subject to the number of running apps, owing to their high computational cost. To mitigate this limitation, we propose DynaMIX (Dynamic MIXed-precision model construction), which optimizes the resource requirement of concurrent apps and aims to maximize execution accuracy. To realize a real-time resource optimization, we formulate an optimization problem using app performance profiles to consider both the accuracy and worst-case latency of each app. We also propose dynamic model reconfiguration by lazy loading only the selected layers at runtime to reduce the overhead of loading the entire model. DynaMIX is evaluated in terms of constraint satisfaction and inference accuracy for a multi-tasking system and compared against state-of-the-art solutions, demonstrating its effectiveness and feasibility under various environmental/operating conditions.
BibTeX
@misc{cho2023dynamix,
  author = {Minkyoung Cho and Kang G. Shin},
  title = {{DynaMIX: Resource Optimization for DNN-Based Real-Time Applications on a Multi-Tasking System}},
  year = {2023},
  eprint = {2302.01568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.01568},
}
Figure