Evaluating Emerging AI/ML Accelerators: IPU, RDU, and NVIDIA/AMD GPUs

Hongwu Peng, Caiwen Ding, Tong Geng, Sutanay Choudhury, Kevin J. Barker, Ang Li

TL;DR

This paper provides a head-to-head performance evaluation of the "new guard" of AI hardware—Graphcore’s Intelligence Processing Unit (IPU) and SambaNova’s Reconfigurable Dataflow Unit (RDU)—against industry-standard NVIDIA and AMD GPUs. The study reveals that while GPUs remain the most versatile, dataflow architectures like the IPU can outperform them by over 8x in specific workloads like 2D convolutions, though they face challenges with compiler maturity and memory constraints.

What Problem Does This Paper Address?

For decades, the von Neumann architecture (where CPU and memory are separate) has been the bedrock of computing. However, modern AI/ML workloads—characterized by massive matrix operations and complex data movement—are hitting a "memory wall" in these traditional designs.

To solve this, companies have developed specialized accelerators. Some, like NVIDIA and AMD, have evolved the GPU with specialized "Tensor Cores." Others, like Graphcore and SambaNova, have built entirely new "dataflow" architectures that aim to minimize data movement by keeping computation as close to the data as possible. This paper seeks to provide an objective, empirical comparison of these diverse architectures to help researchers understand which hardware is best suited for specific AI tasks.

Methods

The authors performed a "bottom-up" benchmark evaluation. Instead of just running full models, they isolated the fundamental building blocks (operators) of modern AI:

  • GEMM (General Matrix Multiplication): The core of almost every neural network.
  • 2D Convolutions (Conv2D): The heart of computer vision.
  • SPMM (Sparse-Matrix Dense-Matrix Multiplication): Critical for Graph Neural Networks (GNNs) and pruned models.
  • BERT Operators: Specific attention-mechanism patterns used in Large Language Models.
  • Streaming Operators: Element-wise functions like ReLU and Sigmoid.

They tested these across multiple precisions (FP16, FP32, FP64) on a formidable hardware lineup: Graphcore GC2000 (IPU), SambaNova SN10 (RDU), NVIDIA V100/A100, and AMD MI100/MI250.

Datasets

While the paper focuses on operators rather than training datasets, it utilizes several standard benchmarks for evaluation:

  • CNN Architectures: ResNet-18, ResNet-50, and MobileNetV2 were used to define the parameters for convolution benchmarks.
  • SuiteSparse Library: A collection of 100 real-world sparse matrices used to evaluate SPMM performance.
  • BERT Configurations: Various batch sizes and sequence lengths (e.g., 128 to 512) to simulate transformer workloads.

Key Insights and Ideas

  • The Power of On-Chip Memory: The Graphcore IPU’s massive 900MB of in-processor SRAM allows it to keep entire layers of a model "on-chip." This is why it achieved a staggering 8.18x speedup in FP16 convolutions compared to the NVIDIA V100.
  • Dataflow vs. SIMT: GPUs use a Single Instruction, Multiple Threads (SIMT) model, which is efficient but rigid. The IPU (MIMD) and RDU (CGRA) offer more flexibility for "irregular" workloads, such as sparse matrices or complex graphs, where data doesn't always move in neat, predictable blocks.
  • The "Compiler Gap": A recurring theme is that hardware is only as good as its software. The SambaNova RDU showed "unstable" performance for large matrices because its spatial compiler struggled to map the operations efficiently to the hardware units.
  • Precision Matters: While the A100 dominates in FP16 and FP64 throughput, the IPU GC2000 actually delivered the highest FP32 performance for GEMM operations, making it a strong contender for tasks requiring higher numerical precision.

Key Results

  • GEMM: The NVIDIA A100 achieved the highest throughput for FP16 (10.64x normalized to V100), but the IPU GC2000 led in FP32 (2.55x normalized).
  • Convolutions: The IPU GC2000 was the clear winner, outperforming the A100 by nearly 4x in FP16 and 2.5x in FP32 for CNN-style operators.
  • SPMM (Sparsity): NVIDIA GPUs, using the CSR (Compressed Sparse Row) format, currently hold the performance lead. While the IPU shows "great potential" for sparsity, its performance was less optimized for varying matrix sizes.
  • RDU Performance: The SambaNova SN10 delivered competitive FP16 performance (2.96x normalized) but was hampered by the lack of compiler support for certain operators like SPMM.

Limitations and Future Work

The authors identify several hurdles for the emerging accelerators:

  1. Memory Capacity: The IPU’s reliance on on-chip SRAM means it cannot handle extremely large matrices (beyond ~10,000) as easily as GPUs with their massive HBM (High Bandwidth Memory) stacks.
  2. Software Maturity: The "unstable" results for the RDU and the lack of support for specific sparse formats on the IPU highlight that the software ecosystems for these new chips are still catching up to NVIDIA’s CUDA.
  3. Scalability: This study focused largely on single-chip or small-node performance. Future research is needed to see how these dataflow advantages hold up when scaling to massive clusters of thousands of chips.

Summary updated

Abstract

The relentless advancement of artificial intelligence (AI) and machine learning (ML) applications necessitates the development of specialized hardware accelerators capable of handling the increasing complexity and computational demands. Traditional computing architectures, based on the von Neumann model, are being outstripped by the requirements of contemporary AI/ML algorithms, leading to a surge in the creation of accelerators like the Graphcore Intelligence Processing Unit (IPU), Sambanova Reconfigurable Dataflow Unit (RDU), and enhanced GPU platforms. These hardware accelerators are characterized by their innovative data-flow architectures and other design optimizations that promise to deliver superior performance and energy efficiency for AI/ML tasks.This research provides a preliminary evaluation and comparison of these commercial AI/ML accelerators, delving into their hardware and software design features to discern their strengths and unique capabilities. By conducting a series of benchmark evaluations on common DNN operators and other AI/ML workloads, we aim to illuminate the advantages of data-flow architectures over conventional processor designs and offer insights into the performance trade-offs of each platform. The findings from our study will serve as a valuable reference for the design and performance expectations of research prototypes, thereby facilitating the development of next-generation hardware accelerators tailored for the ever-evolving landscape of AI/ML applications. Through this analysis, we aspire to contribute to the broader understanding of current accelerator technologies and to provide guidance for future innovations in the field.
BibTeX
@inproceedings{peng2024evaluating,
  author = {Hongwu Peng and Caiwen Ding and Tong Geng and Sutanay Choudhury and Kevin J. Barker and Ang Li},
  title = {{Evaluating Emerging AI/ML Accelerators: IPU, RDU, and NVIDIA/AMD GPUs}},
  booktitle = {ICPE (Companion)},
  year = {2024},
  doi = {10.1145/3629527.3651428},
}
Figure