Optimizing Memory-Access Patterns for Deep Learning Accelerators

Hongbin Zheng, Sejong Oh, Huiqing Wang, Preston Briggs, Jiading Gai, Animesh Jain, Yizhi Liu, Rich Heaton, Randy Huang, Yida WangarXiv 2020 · arXiv

Abstract

Modern deep learning (DL) accelerators achieve high peak performance through specialized compute units like systolic arrays. However, the actual throughput is often limited by the "memory tax"—the overhead of moving data between memory hierarchies. This paper discusses a compiler optimization strategy developed by AWS researchers for the Inferentia accelerator. By utilizing the polyhedral model and global memory-bank mapping, the proposed approach reduces on-chip data copies by up to 76%, shifting the focus from local operation optimization to global graph-level data flow management.

Introduction

In the pursuit of deep learning performance, hardware development has prioritized raw computational power (TFLOPS). As compute units become faster, the primary bottleneck has shifted to memory bandwidth and data movement. On custom hardware like AWS Inferentia or Google’s TPU, this challenge is exacerbated by the use of software-managed scratchpad memory instead of traditional hardware-managed caches.

In these architectures, the compiler is responsible for the precise staging of data. If the compiler fails to optimize memory access patterns across the entire computation graph, the compute units remain idle while waiting for data. This paper explores how a global view of the neural network, rather than an operation-by-operation approach, can significantly mitigate these inefficiencies.

Methodology

The researchers integrated their optimization framework into the AWS Inferentia compiler toolchain, utilizing Apache TVM for intermediate representation and the polyhedral model for mathematical analysis of loop nests.

1. Data-Movement Elimination

Many operations in deep learning, such as transpose, split, or repeat, are "view-only" operations that do not modify the underlying data values. The researchers represented these operations using affine transformations. By mathematically composing the "load" requirements of a subsequent operation with the "store" requirements of a preceding one, the compiler can identify redundant data movements. In many cases, the compiler can prove that intermediate data does not need to be written back to memory at all, allowing it to be piped directly between compute tasks.

2. Global Memory-Bank Mapping

To enable parallel access, accelerator memory is typically divided into multiple banks. Moving data between these banks is a costly operation. The authors proposed a "fixed-point iteration" algorithm to optimize data placement. The algorithm identifies operations with strict hardware constraints (e.g., a convolution requiring specific input alignment) and propagates these requirements throughout the network graph. This ensures that the output of one layer is stored in the specific memory bank required by the subsequent layer, eliminating the need for shuffles.

Experimental Evaluation

The optimizations were evaluated on Amazon EC2 Inf1 instances featuring the Inferentia chip. Two primary workloads were used to benchmark the system:

  • Parallel WaveNet: Selected to evaluate the effectiveness of data-movement elimination.
  • ResNet-50: Selected to evaluate global memory-bank mapping across a standard computer vision pipeline.

Key Results

The experimental results indicate that global memory optimization provides substantial gains over local, per-operation optimization:

  • Parallel WaveNet: The compiler successfully eliminated 123 out of 124 load-store pairs. This optimization saved 145 MB of intermediate storage and reduced on-chip memory copies by 10%.
  • ResNet-50: By applying global memory-bank mapping, the researchers eliminated 76% of on-chip data copies and 37% of off-chip copies compared to standard local mapping techniques.

Discussion and Conclusion

The findings suggest that local optimization is often a trap in DL compiler design; an operation that appears optimal in isolation may impose significant memory overhead on the next stage of the pipeline. By treating the computation graph as a unified entity and applying formal logic via the polyhedral model, compilers can move away from fragile heuristics toward verified, optimal data flow.

While the current implementation is tailored to the specific architecture of AWS Inferentia, the principle of global constraint propagation is broadly applicable to other accelerators with software-managed memory. Future work should address the computational overhead of these polyhedral transformations and their scalability to larger, more irregular models like Transformers or Graph Neural Networks (GNNs).

Summary updated

BibTeX
@misc{zheng2020optimizing,
  author = {Hongbin Zheng and Sejong Oh and Huiqing Wang and Preston Briggs and Jiading Gai and Animesh Jain and Yizhi Liu and Rich Heaton and Randy Huang and Yida Wang},
  title = {{Optimizing Memory-Access Patterns for Deep Learning Accelerators}},
  year = {2020},
  eprint = {2002.12798},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.12798},
}
Figure