Modeling Deep Learning Accelerator Enabled GPUs

Md Aamir Raihan, Negar Goli, Tor M. Aamodt

TL;DR

This paper dives deep into NVIDIA's Tensor Cores, specialized hardware units for accelerating deep learning. The authors developed a microarchitectural model for these cores and integrated it into the GPGPU-Sim simulator, achieving a remarkable 99.6% correlation with real hardware performance. This work provides crucial insights into how these accelerators function and enables more accurate simulation of deep learning workloads.

What Problem Does This Paper Address?

Deep learning has exploded in popularity, but it's computationally intensive. NVIDIA introduced "Tensor Cores" in their Volta and Turing GPUs to speed up the matrix operations fundamental to deep learning. However, NVIDIA hasn't publicly detailed the internal design of these cores. This lack of transparency makes it difficult for researchers and developers to accurately model and optimize for them. This paper aims to bridge this gap by reverse-engineering the Tensor Core's microarchitecture and creating a functional model for simulation.

Methods

The authors employed a multi-pronged approach:

  • Microbenchmarking: They designed and executed specialized microbenchmarks to analyze how data is transferred to and from Tensor Cores, and to measure the timing of their operations. This involved using tools like radare2 to manipulate machine instructions and observe their effects.
  • Reverse Engineering: By disassembling NVIDIA's machine code (SASS) for Tensor Core operations (WMMA API), they inferred the low-level instructions (HMMA) and how they map to hardware.
  • Architectural Modeling: Based on their observations, they proposed a microarchitectural model for the Volta Tensor Cores, detailing how threads, threadgroups, and octets interact with the core's functional units.
  • Simulator Integration: They extended the GPGPU-Sim simulator to incorporate functional and timing models for Tensor Core operations, including support for half-precision (FP16) floating-point arithmetic. They also added support for the CUTLASS library, a popular tool for using Tensor Cores.

Datasets

The paper does not explicitly mention training datasets. For evaluation, they used:

  • NVIDIA Titan V GPU: Used as the real-world hardware for performance comparison.
  • GEMM (General Matrix Multiply) workloads: These are standard benchmarks for evaluating matrix operations, crucial for deep learning. They used both WMMA-based kernels and kernels developed with the CUTLASS library.

Key Insights and Ideas

  • Tensor Cores are Highly Parallelized: A single WMMA instruction (which operates on larger tiles) is broken down into many smaller HMMA instructions, each performing a 4x4 matrix multiply-accumulate (MACC) operation.
  • Thread Cooperation is Key: Threads within a warp don't work in isolation. They are organized into "threadgroups" and "octets" that cooperatively load data and perform computations, with specific octets responsible for different parts of the result.
  • Data Movement is Complex: The way matrix elements are distributed across threads and registers is intricate, with different patterns for operand matrices A, B, and C, and variations between Volta and Turing architectures.
  • Volta vs. Turing Differences: While both architectures feature Tensor Cores, Turing introduces new precision modes (1-bit, 4-bit, 8-bit) and slightly different instruction sequencing compared to Volta.
  • GPGPU-Sim is a Powerful Tool: The paper demonstrates the effectiveness of GPGPU-Sim for accurately modeling complex hardware features like Tensor Cores, achieving high correlation with real hardware.

Key Results

  • 99.6% IPC Correlation: The modified GPGPU-Sim achieved a 99.6% correlation in Instructions Per Cycle (IPC) with the NVIDIA Titan V GPU when running Tensor Core-enabled GEMM workloads.
  • Performance Gains: Tensor Cores provide significant speedups, with the paper reporting gains of approximately 3-6x over standard single-precision GEMM kernels and about 3x over half-precision GEMM kernels.
  • Latency Measurements: The minimum latencies for WMMA instructions were measured: wmma.load (125 cycles), wmma.store (120 cycles), and wmma.mma (70 cycles) for a 1024x1024 GEMM kernel.
  • CUTLASS Integration: The paper successfully enabled the CUTLASS library to run on GPGPU-Sim, facilitating the study of its performance.

Limitations and Future Work

  • PTX Level Modeling: The current GPGPU-Sim model operates at the PTX (Parallel Thread Execution) instruction set level, not the lower-level SASS (Shader Execution Assembly). While accurate, modeling at SASS could potentially reveal finer-grained details.
  • Limited Turing Modes: The 1-bit mode for Turing Tensor Cores was not fully analyzed due to experimental issues.
  • Future Work: The authors suggest further investigation into the microarchitectural details of Turing Tensor Cores, exploring the impact of different precision modes, and extending the GPGPU-Sim model to support SASS-level analysis. They also highlight the potential for their work to serve as a baseline for future research in deep learning accelerator modeling.

Summary updated

BibTeX
@inproceedings{raihan2019modeling,
  author = {Md Aamir Raihan and Negar Goli and Tor M. Aamodt},
  title = {{Modeling Deep Learning Accelerator Enabled GPUs}},
  booktitle = {ISPASS},
  year = {2019},
  url = {https://openreview.net/forum?id=qNDJaA22rLm},
}

Loading related content...

Figure