TL;DR
This paper presents modifications to the GPGPU-Sim simulator to enable the analysis of PyTorch and cuDNN workloads, bridging a critical gap in GPU simulation for machine learning research. By enabling simulation of these popular frameworks, the authors provide a valuable tool for understanding and optimizing deep learning performance on GPUs.
What Problem Does This Paper Address?
Training modern deep neural networks (DNNs) demands immense computational power, primarily provided by GPUs. While GPU hardware has rapidly evolved to support machine learning (e.g., Tensor Cores), academic research has lagged in optimizing GPUs for these tasks, partly due to a lack of simulators capable of running popular machine learning frameworks like PyTorch and TensorFlow. These frameworks rely on specialized, often proprietary, libraries like cuDNN, whose internal workings are not fully documented, making them difficult to simulate with existing open-source tools. This paper addresses this gap by extending GPGPU-Sim to accurately simulate cuDNN-based workloads, thereby enabling the study of PyTorch and similar frameworks.
Methods
The core contribution is the modification of the GPGPU-Sim simulator to support cuDNN. This involved several key steps:
- Enabling cuDNN Execution: The primary challenge was allowing GPGPU-Sim to execute PTX kernels embedded within NVIDIA's cuDNN library. This was achieved by modifying the simulator to handle dynamically linked libraries and to process PTX files that might have duplicate definitions across multiple source files.
- CUDA API Support: Additional CUDA Runtime and API functions, such as
cudaStreamWaitEventand thebrev.type d, a;PTX instruction, were implemented to support cuDNN's specific usage patterns. - Debugging Functional Simulation Errors: A significant effort was dedicated to developing a robust debugging methodology. This involved a "differential coverage analysis" approach, comparing simulator execution with hardware results to pinpoint incorrect instructions. A novel debugging tool was created to systematically identify the first incorrect instruction within a failing kernel.
- Checkpointing Support: To handle the long simulation times of machine learning workloads, checkpointing functionality was added to GPGPU-Sim. This allows users to save simulation state and resume later, significantly improving usability for performance analysis.
- FP16 Support (Partial): Basic support for 16-bit floating-point (FP16) operations was added, though simulating the nuances of fused multiply-add (FMA) instructions for FP16 remains a future work.
- PyTorch and TensorFlow Integration: Efforts were made to enable PyTorch execution by addressing issues with dynamic linking and library paths. Initial steps were also taken towards TensorFlow support, though full integration is left for future work.
Datasets
The primary dataset used for evaluation and correlation was MNIST, a standard benchmark for image classification. The paper specifically mentions using a cuDNN-enabled implementation of LeNet trained on MNIST.
Key Insights and Ideas
- The Simulation Gap is Real: The paper highlights a significant disconnect between the capabilities of existing GPU simulators and the demands of modern machine learning frameworks, particularly those leveraging vendor-specific libraries like cuDNN.
- cuDNN's Complexity: cuDNN's internal implementation, often using hand-tuned assembly and complex API calls, presents substantial challenges for accurate simulation.
- Debugging is Crucial: Simulating complex workloads like those in deep learning requires sophisticated debugging tools. The paper's novel approach to identifying the first incorrect instruction is a valuable contribution.
- Microarchitecture Insights from Simulation: Even with some discrepancies, the modified GPGPU-Sim can reveal performance bottlenecks and microarchitectural behaviors, such as DRAM partition bank camping, that are not easily observable otherwise.
- Checkpointing is Essential for ML Workloads: The extremely long runtimes of ML training/inference on simulators necessitate checkpointing for practical analysis.
Key Results
- Correlation with Hardware: The modified GPGPU-Sim achieved a 72% correlation with real hardware (GeForce GTX 1050) for a 32-bit floating-point version of MNIST. Per-kernel correlations showed larger discrepancies for kernels like CGEMM, Winograd, and LRN, indicating areas for future improvement.
- Identification of Performance Issues: The simulator was used to observe potentially inefficient microarchitecture behavior, such as DRAM partition bank camping, within cuDNN API calls.
- Debugging Tool: A new debugging tool was developed and used to successfully identify functional simulation errors, enabling correct execution of the MNIST workload.
Limitations and Future Work
- Tensor Core Support: The current GPGPU-Sim modifications do not support NVIDIA's Tensor Cores, as cuDNN's PTX code does not directly expose these operations, and the simulator's SASS support is limited to older GPU generations.
- FP16 Simulation Nuances: While FP16 support was added, accurately simulating the precision and rounding behavior of FMA instructions for FP16 remains a challenge for future work.
- TensorFlow Integration: Full support for TensorFlow, particularly its array initialization syntax, is left as future work.
- Double Pointer Arguments: Enhancing the simulator to handle double pointer arguments in kernel inputs is also identified as future work.
- Deeper Microarchitectural Analysis: The authors suggest that further analysis using the simulator could reveal more optimization opportunities, such as reducing idle power consumption by turning off cores during inactive phases.
Summary updated