TL;DR
This paper tackles the critical issue of simulator accuracy in GPU architecture research. By developing a significantly more detailed memory system model for GPGPU-Sim, the authors demonstrate that existing simulators can lead to flawed architectural design decisions, overstating the impact of some optimizations while underestimating others. Their enhanced model achieves much higher correlation with real NVIDIA Volta hardware, offering a more reliable tool for future GPU research.
What Problem Does This Paper Address?
Designing efficient GPUs is a complex endeavor, and architects heavily rely on simulators to evaluate new ideas before committing to costly hardware implementations. However, the accuracy of these simulators is paramount. As GPUs become increasingly complex, especially their memory systems, high-level simulators can introduce significant errors. These inaccuracies can lead to two major problems: discarding promising architectural ideas prematurely because they appear ineffective in a simplified model, or wasting time and resources on ineffective designs that seem beneficial due to simulation artifacts. This paper specifically addresses the gap between the simplified memory models in popular GPU simulators like GPGPU-Sim and the intricate reality of modern GPU memory systems, particularly the NVIDIA Volta architecture.
Methods
The core of this work is the creation of an enhanced memory system model for the GPGPU-Sim simulator. Instead of simply scaling existing configurations, the authors performed a module-by-module redesign, incorporating detailed insights gained from extensive micro-benchmarking and analysis of the NVIDIA Volta architecture. Key enhancements include:
- Detailed Volta Memory Coalescer: Modeling the Volta architecture's specific coalescing behavior, which differs from older architectures.
- Streaming, Banked, Sectored L1 Cache: Implementing a more accurate L1 cache model that accounts for its streaming nature, banking, and sectoring, along with an "allocate-on-fill" policy.
- Volta L2 Cache Policy: Accurately modeling the L2 cache's write policy, specifically a "lazy fetch-on-read" approach that balances write-validate and fetch-on-write.
- Advanced Memory Partitioning: Incorporating techniques to reduce uneven memory accesses across partitions.
- HBM Modeling: Accurately simulating High Bandwidth Memory, including its dual-bus interface and per-bank refresh commands.
- CPU-GPU Memory Copy Engine: Modeling this crucial component, which significantly impacts L2 accesses.
To validate their model, they developed a "Correlator" toolset that includes a suite of benchmarks and a database of hardware profiling results.
Datasets
The paper utilizes a comprehensive set of workloads for evaluation, including applications from popular GPGPU benchmark suites such as Rodinia, CUDA SDK, Parboil, Pannotia, Lonestar, SHOC, and DOE proxy apps. These workloads were compiled with CUDA 10 and targeted the Volta architecture's compute capability (sm_70). The primary "dataset" for validation, however, is the real hardware performance data collected from an NVIDIA TITAN V GPU using tools like nvprof.
Key Insights and Ideas
- Simulator Accuracy Drastically Alters Design Decisions: The most striking insight is how a more accurate memory model can flip the perceived effectiveness of architectural features. Out-of-order memory scheduling, which appears less impactful in simplified models, becomes significantly beneficial with a detailed memory model. Conversely, the L1 cache throughput bottleneck, a common focus in older research, is largely mitigated in modern Volta GPUs due to advanced techniques, making it less of a critical design concern.
- Modern GPUs Have Solved the L1 Bottleneck: The paper highlights that contemporary GPUs like Volta employ sophisticated techniques (sectored, banked, streaming L1s) that effectively eliminate the L1 cache throughput bottleneck that was prevalent in earlier architectures.
- Memory System Details Matter Immensely: The research underscores that even seemingly minor details in the memory system, such as L1 cache sectoring, adaptive policies, and specific L2 write behaviors, have a substantial impact on simulation accuracy and, consequently, on architectural design choices.
- The Need for Continuous Simulator Updates: As GPU hardware evolves rapidly, academic simulators must keep pace. Relying on scaled-down models of older architectures (like Fermi) for modern GPUs leads to significant inaccuracies.
Key Results
- Reduced Execution Time Error: The enhanced GPGPU-Sim model reduces execution time error by 2.5x compared to the older model when simulating an NVIDIA TITAN V GPU.
- Reduced Memory System Counter Error: The error in memory system counters is reduced by as much as 66x.
- High Correlation with Hardware: The new model achieves significantly higher correlation with real hardware across various metrics, including L1/L2 read/write transactions, L1 hit rates, and DRAM read transactions. For instance, L2 read transaction correlation improves from 48.71% error to 1.15% error.
- Demonstrated Impact on Design Choices: The paper provides concrete examples showing how architectural features like out-of-order memory scheduling become more important with the accurate model, while L1 cache contention becomes less so.
Limitations and Future Work
- Limitations: The authors acknowledge that even with their enhanced model, some pervasive error still exists in certain applications. They attribute this to potential complexities in the warp scheduling policy and L1 eviction policy, which would require even more detailed reverse engineering. They also note that their correlation for L1 hit rate is somewhat challenging due to profiler inconsistencies.
- Future Work: The paper introduces the "Correlator" toolset, which is intended to be released publicly. This toolset aims to facilitate future validation efforts and the development of more accurate simulation models. The authors hope their work will encourage the academic community to invest in more detailed simulation models to keep pace with industrial hardware advancements. They also suggest that the improved accuracy opens up new avenues for research into advanced memory access scheduling.
Summary updated