Introduction
Autonomous vehicles (AVs) rely on a suite of Deep Neural Network (DNN) applications—such as object detection, lane tracking, and sign classification—running simultaneously. These applications share a common "heartbeat": the camera's frame rate. If a camera captures 30 frames per second, every application must finish processing the current frame before the next one arrives.
The challenge is that onboard resources (CPU, GPU, memory) are limited. When the number of active apps changes—for instance, when a car enters a busy intersection and triggers more specialized detection tasks—the system can become overloaded. Standard multi-tasking systems often miss deadlines because they treat DNN models as static "black boxes."
The DynaMIX Framework
The authors propose DynaMIX (Dynamic MIXed-precision model construction), a resource optimization framework designed to ensure concurrent deep learning applications meet strict real-time deadlines. By combining mixed-precision model construction with a "lazy loading" mechanism, it maximizes execution accuracy while preventing the "deadline misses" that occur when multiple heavy models compete for limited onboard compute.
1. Offline Profiling and Model Building
For each application, the system builds a library of mixed-precision models. It identifies which layers are most sensitive to quantization (using KL-Divergence) and keeps them in high precision (FP32) while quantizing less sensitive layers to INT8. It then generates "performance profiles"—mathematical curves (cubic polynomials) that predict the latency and accuracy of a model based on its memory footprint.
2. Online Resource Allocation
When the workload changes, a main process solves a constrained nonlinear optimization problem. It finds the optimal memory allocation for each app that maximizes total system accuracy while ensuring the sum of all app latencies stays below the frame-rate deadline. By using polynomial regression to create "profiles" offline, the online controller can make complex allocation decisions in milliseconds.
3. Lazy Loading Reconfiguration
To avoid the massive overhead of swapping entire models in and out of memory, DynaMIX uses "lazy loading." It keeps a base INT8 version of the model in memory and only loads the specific high-precision (FP32) layers required for the newly selected configuration at runtime. This allows the system to reconfigure a model's precision on-the-fly without stalling the execution pipeline.
Experimental Evaluation
The framework was evaluated using three representative DNN models and their corresponding standard benchmarks: VGG16 (CIFAR-10), ResNet50 (ImageNet), and YOLO-v3 (MS COCO).
Key results include:
- Deadline Satisfaction: In scenarios where a standard baseline would miss deadlines by hundreds of milliseconds, DynaMIX successfully reconfigured models to stay within the 250ms–700ms target windows.
- High Accuracy: Despite aggressive compression, the total accuracy drop across all concurrent apps was kept under 2.2%.
- Efficiency Gains: For the YOLO-v3 model, the most compressed version reduced latency by 4.3x and memory usage by 1.5x compared to the full-precision baseline.
- Comparison to State-of-the-Art: Compared to NestDNN (a leading pruning-based approach), DynaMIX achieved significantly higher inference accuracy (92.5% vs 89% for VGG16) because it optimizes for mixed-precision rather than just removing filters.
Conclusion and Future Work
DynaMIX demonstrates that sensitivity-aware mixed-precision is a powerful tool for real-time resource management. Future work will explore relaxing the assumption that all base models must fit in memory simultaneously, incorporating ultra-low precision (INT4/INT2), and implementing "mixed criticality" modes to prioritize safety-critical tasks like emergency braking over less critical functions.
Summary updated