ME- ViT: A Single-Load Memory-Efficient FPGA Accelerator for Vision Transformers

Kyle Marino, Pengmiao Zhang, Viktor K. PrasannaHiPC 2023 · DOI · OpenReview

TL;DR

ME-ViT is a novel FPGA accelerator that smashes the "memory wall" for Vision Transformers (ViTs) by enforcing a strict single-load policy. By keeping model parameters and intermediate results on-chip and using a versatile, multi-purpose processing element, it achieves up to a 17.89× reduction in memory bandwidth and a 5.10× throughput boost over existing state-of-the-art FPGA designs.

What Problem Does This Paper Address?

Vision Transformers have revolutionized computer vision, but they are notoriously "heavy." Their massive parameter counts and complex attention mechanisms create a significant hardware challenge: memory bandwidth bottlenecking.

In most current systems (including GPUs and standard FPGA accelerators), the processor spends more time waiting for data to move from off-chip memory (DRAM) than it does actually performing calculations. This is especially problematic for "edge" devices where power is limited and bandwidth is narrow. Existing accelerators often reload weights multiple times or send intermediate data back to a host CPU for non-linear operations like Softmax, leading to "stalls" and wasted energy.

Methods

The authors introduce ME-ViT, built around two core pillars:

  1. The Single-Load Policy: Parameters are loaded from DRAM exactly once per inference. Intermediate results are stored in on-chip buffers (BRAM/LUTRAM) rather than being written back to external memory.
  2. The ME-PE (Memory-Efficient Processing Element): Instead of having separate hardware for different parts of the Transformer (Linear Projection, Self-Attention, and MLP), the authors designed a single, highly flexible PE.
  3. Multi-purpose Buffers: These buffers are strategically repurposed across different stages of the computation to minimize the total amount of on-chip memory needed.
  4. Integrated Non-linearities: Hardware-optimized versions of LayerNorm and Pseudo-Softmax (using base-2 instead of base-e for easier hardware implementation) are pipelined directly into the matrix multiplication flow. This eliminates the need to communicate with a host CPU.
  5. DSP Packing: The design uses a technique to perform two simultaneous multiplications per DSP (Digital Signal Processing) slice, effectively doubling the math throughput of the FPGA's hardware.

Datasets

The researchers evaluated ME-ViT using four standard Vision Transformer variants:

  • ViT-Base (ViT-B): The original standard model.
  • DeiT-B, DeiT-S, and DeiT-T: Data-efficient Image Transformers in Base, Small, and Tiny configurations.

These models were tested on a Xilinx Alveo U200 FPGA, a high-performance data center card.

Key Insights and Ideas

  • Communication is More Expensive than Computation: The "aha moment" here is that optimizing how data moves is more impactful than simply adding more raw math power. By focusing on the "Compute-to-Communication" (C2C) ratio, they reached near-peak performance on the roofline model.
  • Strategic Resource Reuse: A Transformer block has a predictable sequence. By designing the ME-PE to "morph" its buffer usage as it moves from the Attention phase to the MLP phase, the authors could fit a full-scale accelerator into the limited on-chip memory of the FPGA.
  • Eliminating CPU Round-Trips: Many accelerators offload "messy" math like Softmax to the CPU. ME-ViT keeps everything on the FPGA, which prevents the entire pipeline from stalling while waiting for the PCIe bus.

Key Results

  • Memory Bandwidth: Achieved a 9.22× to 17.89× improvement in memory bandwidth efficiency compared to unoptimized baselines.
  • Throughput: On the Alveo U200, a 5-PE configuration delivered a 5.10× improvement in throughput (frames per second) over previous state-of-the-art FPGA accelerators.
  • Efficiency:
    • 2.16× better throughput-per-DSP than previous designs.
  • Up to 5.85× better power efficiency (FPS/Watt) compared to a high-end GPU (Titan RTX).
  • Up to 1.51× better power efficiency compared to existing FPGA-based ViT accelerators.

Limitations and Future Work

  • On-Chip Memory Scaling: While the single-load policy is powerful, it is limited by the FPGA's on-chip BRAM. For massive models like ViT-Huge, the parameters might exceed the capacity of even large FPGAs, requiring a "tiled" approach that might break the strict single-load rule.
  • Fixed-Point Quantization: The study uses 8-bit quantization for weights. While standard in the field, very sensitive tasks might require higher precision, which would increase memory pressure.
  • Future Directions: The authors suggest extending this architecture to Large Language Models (LLMs), which share the Transformer architecture and suffer from even more severe memory bottlenecks.

Summary updated

Abstract

Vision Transformers (ViTs) have emerged as a state-of-the-art solution for object classification tasks. However, their computational demands and high parameter count make them unsuitable for real-time inference, prompting the need for efficient hardware implementations. Existing hardware accelerators for ViTs suffer from frequent off-chip memory access, restricting the achievable throughput by memory bandwidth. In devices with a high compute-to-communication ratio (e.g., edge FPGAs with limited bandwidth), off-chip memory access imposes a severe bottleneck on overall throughput. This work proposes ME-ViT, a novel Memory Efficient FPGA accelerator for ViT inference that minimizes memory traffic. We propose a single-load policy in designing ME-ViT: model parameters are only loaded once, intermediate results are stored on-chip, and all operations are implemented in a single processing element. To achieve this goal, we design a memory-efficient processing element (ME-PE), which processes multiple key operations of ViT inference on the same architecture through the reuse of multi-purpose buffers. We also integrate the Softmax and LayerNorm functions into the ME-PE, minimizing stalls between matrix multiplications. We evaluate ME-ViT on systolic array sizes of 32 and 16, achieving up to a 9.22x and 17.89x overall improvement in memory bandwidth, and a 2.16 x improvement in throughput per DSP for both designs over state-of-the-art ViT accelerators on FPGA. ME-ViT achieves a power efficiency improvement of up to 4.00 x (1.03x) over a GPU (FPGA) baseline. ME-ViT enables up to 5 ME-PE instantiations on a Xilinx Alveo U200, achieving a 5.10 x improvement in throughput over the state-of-the art FPGA baseline, and a 5.85 x (1.51 x) improvement in power efficiency over the GPU (FPGA) baseline.
BibTeX
@inproceedings{marino2023me,
  author = {Kyle Marino and Pengmiao Zhang and Viktor K. Prasanna},
  title = {{ME- ViT: A Single-Load Memory-Efficient FPGA Accelerator for Vision Transformers}},
  booktitle = {HiPC},
  year = {2023},
  doi = {10.48550/arxiv.2402.09709},
}
Figure