Abstract
Modern AI/ML workloads are increasingly hitting a "memory wall" in traditional von Neumann architectures. This paper evaluates the performance of "new guard" accelerators—Graphcore’s Intelligence Processing Unit (IPU) and SambaNova’s Reconfigurable Dataflow Unit (RDU)—against established NVIDIA and AMD GPUs. Through a bottom-up benchmarking of fundamental operators, we find that while GPUs offer the most mature software ecosystem, dataflow architectures provide significant performance advantages in specific domains like computer vision, with the IPU achieving over 8x speedup in convolutions.
Introduction: Beyond von Neumann
For decades, the separation of CPU and memory has defined computing. However, the massive data movement required by modern neural networks has exposed the limitations of this design. In response, specialized hardware has emerged. NVIDIA and AMD have evolved GPUs with specialized "Tensor Cores," while companies like Graphcore and SambaNova have pioneered "dataflow" architectures. These new designs aim to minimize data movement by keeping computation as close to the data as possible, often utilizing massive on-chip memory.
Methodology
The evaluation employs a "bottom-up" approach, isolating fundamental AI operators rather than just full-model benchmarks. Key operators tested include:
- GEMM (General Matrix Multiplication): The foundational operation for most neural networks.
- 2D Convolutions (Conv2D): Essential for computer vision tasks.
- SPMM (Sparse-Matrix Dense-Matrix Multiplication): Critical for Graph Neural Networks (GNNs).
- BERT and Streaming Operators: Patterns specific to Transformers and element-wise functions.
The hardware lineup includes the Graphcore GC2000 (IPU), SambaNova SN10 (RDU), NVIDIA V100/A100, and AMD MI100/MI250, tested across FP16, FP32, and FP64 precisions.
Performance Analysis
General Matrix Multiplication (GEMM)
GEMM performance is a primary indicator of an accelerator's raw throughput. The NVIDIA A100 demonstrated the highest throughput for FP16 operations. However, the Graphcore IPU GC2000 led the field in FP32 performance, achieving a 2.55x speedup normalized to the V100. This makes the IPU a strong candidate for scientific computing or tasks requiring higher numerical precision.
2D Convolutions
The most striking results were observed in convolutional workloads. The IPU GC2000 outperformed the A100 by nearly 4x in FP16 and 2.5x in FP32 for CNN-style operators. This performance is largely attributed to the IPU's 900MB of in-processor SRAM, which allows entire model layers to remain on-chip, drastically reducing the energy and time costs of data movement.
Architectural Insights and the "Compiler Gap"
The study highlights the trade-offs between the Single Instruction, Multiple Threads (SIMT) model used by GPUs and the more flexible Multiple Instruction, Multiple Data (MIMD) or Coarse-Grained Reconfigurable Architecture (CGRA) models used by the IPU and RDU. While the latter offer better handling of "irregular" workloads like sparse matrices, they are currently hampered by a "compiler gap." For instance, the SambaNova RDU showed unstable performance for large matrices, suggesting that its spatial compiler still requires optimization to efficiently map operations to hardware units.
Limitations and Conclusion
Despite their performance potential, emerging accelerators face significant hurdles. The IPU’s reliance on on-chip SRAM limits its ability to handle extremely large matrices compared to GPUs equipped with High Bandwidth Memory (HBM). Furthermore, the software maturity of CUDA remains a formidable barrier to entry for new architectures.
In conclusion, while NVIDIA GPUs remain the most versatile and software-mature option, specialized dataflow accelerators like the IPU offer compelling performance benefits for specific workloads, particularly those that can fit within on-chip memory constraints. Future developments in compiler technology and memory scaling will be critical for these "new guard" architectures to achieve broader adoption.
Summary updated