SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts

Raghu Prabhakar, Ram Sivaramakrishnan, Darshan Gandhi, Yun Du, Mingran Wang, Xiangyu Song, Kejie Zhang, Tianren Gao, Angela Wang, Xiaoyan Li, Yongning Sheng, Joshua Brot, Denis Sokolov, Apurv Vivek, Calvin Leung, Arjun Sabnis, Jiayu Bai, Tuowen Zhao, Mark Gottscho, David Jackson, Mark Luttrell, Manish K. Shah, Zhengyu Chen, Kaizhao Liang, Swayambhoo Jain, Urmish Thakker, Dawei Huang, Sumti Jairath, Kevin J. Brown, Kunle Olukotun

Abstract

As Large Language Models (LLMs) scale toward trillion-parameter architectures, traditional GPU-based systems face a critical bottleneck known as the "Memory Wall." While computational throughput has increased exponentially, memory bandwidth and capacity have lagged, leading to hardware underutilization and prohibitive deployment costs. This paper introduces the SambaNova SN40L Reconfigurable Dataflow Unit (RDU), a hardware-software co-design that utilizes a three-tier memory system and a "Composition of Experts" (CoE) framework. By rethinking data movement and utilizing up to 1.5 TiB of local DDR memory, the SN40L achieves a 3.7× speedup over NVIDIA DGX H100 systems for trillion-parameter workloads while reducing the physical hardware footprint by 19×.

1. Introduction: The Memory Wall and the Cost of Monoliths

The AI industry is currently caught between the demand for massive, trillion-parameter models and the physical limitations of existing hardware. Standard GPUs utilize a Single Instruction, Multiple Threads (SIMT) model, which is highly efficient for dense, uniform computations but struggles with the sparse, irregular data patterns often found in modern AI.

Two primary issues have emerged:

  1. The Memory Wall: Accelerators can process data much faster than they can fetch it from memory, leading to "starvation" where the compute units sit idle.
  2. The Hosting Problem: Monolithic models like GPT-4 are too large to fit into the High Bandwidth Memory (HBM) of a single GPU. Even Mixture of Experts (MoE) models, which activate only a fraction of their parameters at a time, still require the entire model to be resident in memory to avoid extreme latency during expert switching.

2. The SN40L Reconfigurable Dataflow Unit (RDU)

To address these bottlenecks, SambaNova developed the SN40L RDU. Unlike the fixed-instruction architecture of a GPU, the RDU is a mesh of Pattern Compute Units (PCUs) and Pattern Memory Units (PMUs). The hardware is dynamically reconfigured to match the specific dataflow graph of the model being executed.

This approach enables Aggressive Operator Fusion. By fusing hundreds of operations into a single kernel, the RDU keeps intermediate data on-chip, minimizing the need for expensive round-trips to global memory. This flexibility also allows the RDU to handle complex data access patterns—such as transposes and shuffles—without the performance penalties associated with SIMT architectures.

3. Three-Tier Memory Hierarchy

The SN40L is the first commercial accelerator to integrate three distinct memory tiers directly into its architecture, providing a balance between speed and capacity:

  • On-chip SRAM (520 MiB): Provides ultra-fast access for intermediate results and fused kernels.
  • On-package HBM (64 GiB): Stores active model weights and Key-Value (KV) caches for high-speed inference.
  • Off-package DDR DRAM (up to 1.5 TiB): This massive capacity allows the system to store hundreds of specialized "expert" models locally.

By attaching DDR directly to the accelerator, the SN40L can swap model weights into HBM in milliseconds, bypassing the traditional CPU-to-GPU bottleneck.

4. Composition of Experts (CoE)

Leveraging this hardware, the authors propose Samba-CoE, a modular software approach. Instead of one monolithic model, the system utilizes a "society" of specialized experts (e.g., 150 experts based on Llama2-7B). A router model directs queries to the most relevant expert.

Because the SN40L node can hold 1.5 TiB of weights in its local DDR, it can host all 150 experts on a single node. In contrast, a GPU-based system would need to spread these experts across 19 different nodes to fit them into HBM, incurring massive networking overhead and infrastructure costs.

5. Performance and Results

The SN40L was evaluated against industry-standard benchmarks, including Llama 2/3.1, Mistral, and Bloom. Key findings include:

  • Inference Speed: For CoE workloads, a single SN40L node is 3.7× faster than a DGX H100 and 6.6× faster than a DGX A100.
  • Switching Efficiency: Moving an expert from DDR to HBM on the SN40L is 15× to 31× faster than moving data from a CPU host to a GPU.
  • Footprint Reduction: To achieve the same latency for a 1-trillion parameter CoE model, the SambaNova architecture requires 19× fewer nodes than a traditional H100-based cluster.
  • Kernel Fusion: The RDU achieved 2× to 13× speedups on specific complex kernels (like FlashFFTConv) compared to unfused baselines.

6. Conclusion and Future Work

The SN40L RDU demonstrates that the "Memory Wall" can be overcome through dataflow architecture and multi-tier memory integration. By moving away from the monolithic model paradigm toward a Composition of Experts, SambaNova provides a path toward serving trillion-parameter AI with significantly lower power and space requirements.

Future work will focus on refining the compiler's dynamic bandwidth modeling to handle bursty traffic and optimizing Least Recently Used (LRU) eviction policies for HBM caching as the number of experts scales into the thousands.

Summary updated

BibTeX
@inproceedings{prabhakar2024sambanova,
  author = {Raghu Prabhakar and Ram Sivaramakrishnan and Darshan Gandhi and Yun Du and Mingran Wang and Xiangyu Song and Kejie Zhang and Tianren Gao and Angela Wang and Xiaoyan Li and Yongning Sheng and Joshua Brot and Denis Sokolov and Apurv Vivek and Calvin Leung and Arjun Sabnis and Jiayu Bai and Tuowen Zhao and Mark Gottscho and David Jackson and Mark Luttrell and Manish K. Shah and Zhengyu Chen and Kaizhao Liang and Swayambhoo Jain and Urmish Thakker and Dawei Huang and Sumti Jairath and Kevin J. Brown and Kunle Olukotun},
  title = {{SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts}},
  booktitle = {MICRO},
  year = {2024},
  doi = {10.1109/micro61859.2024.00100},
}
Figure