TL;DR
This paper tackles the underutilization of heterogeneous computing resources in modern embedded systems like the NVIDIA Xavier SoC, which are increasingly running multiple Deep Neural Network (DNN) instances for tasks like autonomous driving. The authors demonstrate that by creating and optimizing DNN variants for different compute elements (CPU, GPU regular cores, GPU Tensor cores, and NVDLA), they can achieve significant performance gains and improve schedulability, allowing more complex AI functionalities to run concurrently.
What Problem Does This Paper Address?
Modern autonomous driving systems rely heavily on Deep Neural Networks (DNNs) for critical functions like object detection, path planning, and prediction. These systems often run multiple DNN instances simultaneously on a single System-on-Chip (SoC). While SoCs like the NVIDIA Xavier are equipped with diverse computing elements (CPUs, GPU cores, Tensor cores, and dedicated accelerators like NVDLA), current DNN implementations typically only leverage a single type of core (often the GPU's regular cores). This leads to an imbalance in resource utilization, where some compute units are overloaded while others are idle, ultimately increasing task execution times and limiting the overall system's capability. The challenge is to effectively utilize this heterogeneous hardware to meet the demanding real-time requirements of autonomous driving.
Methods
The core of this work involves creating and evaluating specialized variants of DNN libraries for the NVIDIA Xavier SoC. The authors:
- Developed DNN Variants: They implemented different versions of key DNNs used in the Apollo autonomous driving framework, tailoring each variant to specific computing elements (CEs):
- CPU: Variants optimized for 2, 4, or 6 cores.
- GPU Regular Cores: Variants utilizing 4 or 8 Streaming Multiprocessors (SMs).
- GPU Tensor Cores: Variants also utilizing 4 or 8 SMs, specifically leveraging Tensor Cores for matrix operations.
- NVDLA: Variants designed to run on one or two NVIDIA Deep Learning Accelerators.
- Performance Analysis: They conducted extensive timing analysis to characterize the performance and resource utilization of each DNN variant on its respective CE.
- Schedulability Modeling: They employed a Linear Programming (LP) model to represent a static cyclic executive scheduler. This model was used to assess the schedulability of multiple DNN variants running concurrently on the heterogeneous platform, demonstrating how the diverse CE/TLP configurations improve the system's ability to meet deadlines.
Datasets
The paper uses the Apollo autonomous driving software framework as its basis. The authors utilized "Apollo default input data sets," which are described as real data collected from sensors of an autonomous driving car and provided by the Apollo team. They also used "similar neural network architectures that Apollo employs in its different stages." No new benchmarks were introduced; rather, the focus was on analyzing the performance of existing DNNs within the Apollo framework on different hardware.
Key Insights and Ideas
- DNNs are Resource Hogs, but Not Uniformly: Different DNNs have vastly different computational needs and temporal constraints. This diversity is key to exploiting heterogeneous hardware.
- "One Size Fits All" is Inefficient: Relying solely on GPU regular cores for all DNN tasks leaves significant compute resources (Tensor Cores, NVDLA) underutilized, leading to performance bottlenecks.
- Specialization Pays Off: Tailoring DNN implementations to specific compute elements (e.g., using Tensor Cores for matrix-heavy operations, NVDLA for inference) yields substantial performance improvements.
- Heterogeneity Enables More Functionality: By effectively utilizing diverse CEs, systems can support a greater number of concurrent DNN instances, enabling more complex AI functionalities within the same hardware footprint.
- Scheduling is Crucial: A static scheduler that can intelligently map DNN variants to appropriate CEs is essential for maximizing schedulability and meeting real-time deadlines in heterogeneous systems.
Key Results
- Performance Gains: The paper demonstrates significant performance improvements across various DNN instances when using specialized variants on different CEs. For example, NVDLA often provides the best performance for object detection tasks, while Tensor Cores generally outperform regular GPU cores for matrix-intensive operations.
- Variability in Performance: The performance differences between CEs and variants can be substantial, with some DNNs showing orders of magnitude improvement when moved from CPU to GPU.
- Tensor Core Nuances: While Tensor Cores offer significant speedups for large matrix operations, they can sometimes perform worse than regular GPU cores for very small matrix dimensions (N < 12), as observed in the object tracker (M1.t) workload.
- RNN Limitations: Recurrent Neural Networks (RNNs) are noted to not benefit as much from increased parallelization as other DNN types, and some key RNN layers are not yet supported by NVDLA via TensorRT, limiting their deployment on that accelerator.
- Improved Schedulability: The LP-based scheduling analysis shows that by supporting diverse CE/TLP configurations, the system can achieve significantly higher schedulability ratios, allowing more task sets to meet their deadlines, especially under increased utilization thresholds. For instance, in the NVDLA scenario, enabling other CEs allows sustaining workloads up to 2.8x the NVDLA's capacity, compared to only 1.6x with NVDLA alone.
Limitations and Future Work
- RNN Support on NVDLA: The current implementation faces limitations with RNNs on NVDLA due to unsupported layers in TensorRT. Future work could involve adapting or implementing these layers.
- Multi-CE Variants: The current approach assigns each DNN instance to a single CE. The authors suggest future work to explore multi-CE capabilities, where a single instance could leverage multiple CEs simultaneously for even greater flexibility and performance.
- Contention Effects: While the experiments account for contention effects to some extent, the authors assume observed timing results already factor them in. More explicit modeling and analysis of contention in heterogeneous environments could be beneficial.
- Accuracy Trade-offs: The paper briefly mentions that different implementations might use different floating-point standards, potentially leading to minor numerical variations. While functional results matched, a deeper analysis of accuracy trade-offs across CEs could be valuable.
- Dynamic Scheduling: The current work focuses on static scheduling. Exploring dynamic scheduling strategies for heterogeneous platforms could offer further advantages.
Summary updated