AI and Memory Wall

Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer

Abstract

The availability of unprecedented unsupervised training data, along with neural scaling laws, has resulted in an unprecedented surge in model size and compute requirements for serving and training Large Language Models (LLMs). However, the main performance bottleneck is increasingly shifting to memory bandwidth. Over the past 20 years, peak server hardware FLOPS has been scaling at 3.0x every 2 years, significantly outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6x and 1.4x every 2 years, respectively. This disparity has made memory, rather than compute, the primary bottleneck in AI applications, particularly in serving. This paper analyzes encoder and decoder Transformer models, demonstrating how memory bandwidth has become the dominant bottleneck for decoder models and arguing for a fundamental redesign in model architecture, training, and deployment strategies.

Introduction: The Emergence of the Memory Wall

For decades, the advancement of AI has been synonymous with the increase in peak compute power, measured in floating-point operations per second (FLOPS). However, as models like GPT-4 and Llama 2 push the boundaries of scale, a more insidious constraint has emerged: the "Memory Wall." This refers to the widening gap between the speed of processors and the speed at which data can be moved from memory to those processors.

To quantify this trend, researchers from UC Berkeley, ICSI, and LBNL conducted a historical scaling analysis of hardware and AI models over a 20-year period. By profiling the arithmetic intensity—the ratio of total FLOPs to total memory operations (MOPs)—of various architectures, they identified a critical shift where even the most advanced GPUs are frequently "starved" for data, spending significant cycles waiting for DRAM access rather than performing calculations.

The Widening Gap in Hardware Scaling

The fundamental issue lies in the disparate growth rates of different hardware components. While peak hardware compute has seen a staggering 60,000× increase over the last two decades, memory technologies have not kept pace.

Scaling of peak hardware FLOPS vs. memory and interconnect bandwidth over 20 years.

As shown in the figure above, HW FLOPS scales at approximately 3.0x every two years. In contrast, DRAM bandwidth grows at only 1.6x, and interconnect bandwidth (the speed of communication between chips) lags even further at 1.4x. This exponential divergence means that the "compute-to-bandwidth" ratio is worsening, making it increasingly difficult to utilize the full theoretical performance of modern AI accelerators.

The Explosion of Model Requirements

While hardware growth is slowing relative to historical norms, the demands of state-of-the-art (SOTA) AI models are accelerating. The transition from traditional convolutional neural networks to massive Transformer-based models has created a demand for memory capacity and training compute that far exceeds Moore's Law.

Comparison of Transformer scaling in terms of parameter count (left) and training compute (right) against hardware capabilities.

Transformer parameters are currently growing at a rate of 410x every two years, while GPU memory capacity only increases by 2x in the same timeframe. Similarly, the compute required to train these models is scaling at 750x every two years, dwarfing the 2x growth predicted by Moore's Law. This disparity between increasing model size and stagnant memory growth creates a massive bottleneck for both training and inference.

The Decoder Dilemma

A key insight of this research is the architectural sensitivity to the memory wall. The paper highlights a stark contrast between Encoders (e.g., BERT) and Decoders (e.g., GPT).

  • Encoders process tokens concurrently using matrix-matrix operations, which possess high arithmetic intensity and are generally compute-bound.
  • Decoders generate tokens auto-regressively (one by one), relying on matrix-vector operations. This process has much lower arithmetic intensity, making decoders significantly more vulnerable to memory bandwidth limitations.

Benchmarking reveals that despite having similar total FLOP counts, GPT-2 inference latency is significantly higher than BERT-Base. This is because the decoder is bottlenecked by memory operations (MOPs), which scale poorly with sequence length compared to the compute-bound encoder.

Conclusion and Future Directions

The "Memory Wall" is no longer a theoretical concern; it is the primary factor limiting the deployment and efficiency of modern LLMs. To overcome this, the industry must pivot from a pure focus on FLOPS to a more holistic view of the memory hierarchy.

Potential solutions include:

  1. Hardware Redesign: Developing architectures with more sophisticated, larger cache hierarchies to minimize DRAM access.
  2. Algorithmic Efficiency: Utilizing communication-avoiding algorithms and second-order optimization methods that may reduce the overall data movement required.
  3. Model Compression: Aggressive quantization (e.g., INT4) and pruning are becoming essential not just for saving space, but for reducing the bandwidth required to feed the processor.
  4. Small Language Models (SLMs): Optimizing smaller models to fit entirely within on-chip memory could bypass the memory wall entirely for specific applications.

Without these shifts, the gap between what our processors can calculate and what our memory can provide will continue to hinder the next generation of AI innovation.

Summary updated

Abstract

The availability of unprecedented unsupervised training data, along with neural scaling laws, has resulted in an unprecedented surge in model size and compute requirements for serving/training LLMs. However, the main performance bottleneck is increasingly shifting to memory bandwidth. Over the past 20 years, peak server hardware FLOPS has been scaling at 3.0x/2yrs, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively. This disparity has made memory, rather than compute, the primary bottleneck in AI applications, particularly in serving. Here, we analyze encoder and decoder Transformer models and show how memory bandwidth can become the dominant bottleneck for decoder models. We argue for a redesign in model architecture, training, and deployment strategies to overcome this memory limitation.
BibTeX
@article{gholami2024ai,
  author = {Amir Gholami and Zhewei Yao and Sehoon Kim and Coleman Hooper and Michael W. Mahoney and Kurt Keutzer},
  title = {{AI and Memory Wall}},
  journal = {IEEE Micro},
  year = {2024},
  doi = {10.1109/mm.2024.3373763},
}
Figure