TL;DR
This paper provides a head-to-head performance evaluation of the "new guard" of AI hardware—Graphcore’s Intelligence Processing Unit (IPU) and SambaNova’s Reconfigurable Dataflow Unit (RDU)—against industry-standard NVIDIA and AMD GPUs. The study reveals that while GPUs remain the most versatile, dataflow architectures like the IPU can outperform them by over 8x in specific workloads like 2D convolutions, though they face challenges with compiler maturity and memory constraints.
What Problem Does This Paper Address?
For decades, the von Neumann architecture (where CPU and memory are separate) has been the bedrock of computing. However, modern AI/ML workloads—characterized by massive matrix operations and complex data movement—are hitting a "memory wall" in these traditional designs.
To solve this, companies have developed specialized accelerators. Some, like NVIDIA and AMD, have evolved the GPU with specialized "Tensor Cores." Others, like Graphcore and SambaNova, have built entirely new "dataflow" architectures that aim to minimize data movement by keeping computation as close to the data as possible. This paper seeks to provide an objective, empirical comparison of these diverse architectures to help researchers understand which hardware is best suited for specific AI tasks.
Methods
The authors performed a "bottom-up" benchmark evaluation. Instead of just running full models, they isolated the fundamental building blocks (operators) of modern AI:
- GEMM (General Matrix Multiplication): The core of almost every neural network.
- 2D Convolutions (Conv2D): The heart of computer vision.
- SPMM (Sparse-Matrix Dense-Matrix Multiplication): Critical for Graph Neural Networks (GNNs) and pruned models.
- BERT Operators: Specific attention-mechanism patterns used in Large Language Models.
- Streaming Operators: Element-wise functions like ReLU and Sigmoid.
They tested these across multiple precisions (FP16, FP32, FP64) on a formidable hardware lineup: Graphcore GC2000 (IPU), SambaNova SN10 (RDU), NVIDIA V100/A100, and AMD MI100/MI250.
Datasets
While the paper focuses on operators rather than training datasets, it utilizes several standard benchmarks for evaluation:
- CNN Architectures: ResNet-18, ResNet-50, and MobileNetV2 were used to define the parameters for convolution benchmarks.
- SuiteSparse Library: A collection of 100 real-world sparse matrices used to evaluate SPMM performance.
- BERT Configurations: Various batch sizes and sequence lengths (e.g., 128 to 512) to simulate transformer workloads.
Key Insights and Ideas
- The Power of On-Chip Memory: The Graphcore IPU’s massive 900MB of in-processor SRAM allows it to keep entire layers of a model "on-chip." This is why it achieved a staggering 8.18x speedup in FP16 convolutions compared to the NVIDIA V100.
- Dataflow vs. SIMT: GPUs use a Single Instruction, Multiple Threads (SIMT) model, which is efficient but rigid. The IPU (MIMD) and RDU (CGRA) offer more flexibility for "irregular" workloads, such as sparse matrices or complex graphs, where data doesn't always move in neat, predictable blocks.
- The "Compiler Gap": A recurring theme is that hardware is only as good as its software. The SambaNova RDU showed "unstable" performance for large matrices because its spatial compiler struggled to map the operations efficiently to the hardware units.
- Precision Matters: While the A100 dominates in FP16 and FP64 throughput, the IPU GC2000 actually delivered the highest FP32 performance for GEMM operations, making it a strong contender for tasks requiring higher numerical precision.
Key Results
- GEMM: The NVIDIA A100 achieved the highest throughput for FP16 (10.64x normalized to V100), but the IPU GC2000 led in FP32 (2.55x normalized).
- Convolutions: The IPU GC2000 was the clear winner, outperforming the A100 by nearly 4x in FP16 and 2.5x in FP32 for CNN-style operators.
- SPMM (Sparsity): NVIDIA GPUs, using the CSR (Compressed Sparse Row) format, currently hold the performance lead. While the IPU shows "great potential" for sparsity, its performance was less optimized for varying matrix sizes.
- RDU Performance: The SambaNova SN10 delivered competitive FP16 performance (2.96x normalized) but was hampered by the lack of compiler support for certain operators like SPMM.
Limitations and Future Work
The authors identify several hurdles for the emerging accelerators:
- Memory Capacity: The IPU’s reliance on on-chip SRAM means it cannot handle extremely large matrices (beyond ~10,000) as easily as GPUs with their massive HBM (High Bandwidth Memory) stacks.
- Software Maturity: The "unstable" results for the RDU and the lack of support for specific sparse formats on the IPU highlight that the software ecosystems for these new chips are still catching up to NVIDIA’s CUDA.
- Scalability: This study focused largely on single-chip or small-node performance. Future research is needed to see how these dataflow advantages hold up when scaling to massive clusters of thousands of chips.
Summary updated