AutoWS: Automate Weights Streaming in Layer-wise Pipelined DNN Accelerators

Zhewen Yu, Christos-Savvas BouganisarXiv 2023 · DOI · arXiv

Abstract

AutoWS is an automated framework designed to enable weight streaming for layer-wise pipelined Deep Neural Network (DNN) accelerators on FPGAs. By intelligently partitioning weights between on-chip and off-chip memory, AutoWS allows large-scale neural networks to be deployed on resource-constrained devices that previously lacked the capacity to house them. Experimental results demonstrate that AutoWS can significantly reduce on-chip memory usage and latency, often outperforming traditional sequential execution models and industry-standard implementations.

Introduction: The Memory Wall in FPGA Acceleration

In FPGA-based DNN acceleration, designers typically choose between two primary architectures: sequential execution, where a single engine processes layers one by one, and layer-wise pipelining, where each layer is assigned a dedicated hardware engine.

While pipelining offers high efficiency and customization, it faces a significant "memory wall." Conventional pipelined designs attempt to store all model weights in the FPGA's on-chip Block RAM (BRAM). For large-scale models like ResNet50, the weight requirements exceed the BRAM capacity of most mid-range FPGAs. This limitation often forces a retreat to sequential execution, which suffers from high latency due to frequent off-chip data movement. AutoWS bridges this gap by enabling pipelined architectures to stream weights dynamically from off-chip DRAM.

Comparison of execution models: Sequential, Vanilla Pipelining, and AutoWS.

Methodology

AutoWS introduces a three-pronged approach to automate weight streaming:

  1. Hybrid Weight Storage: Rather than an "all-or-nothing" approach, AutoWS partitions weight storage into static regions (permanently on-chip) and dynamic regions (reloaded from off-chip DRAM as needed). This configuration is optimized on a per-layer basis to balance BRAM usage and bandwidth requirements.
  2. Greedy Design Space Exploration (DSE): Determining the optimal weight partition is a complex optimization problem. AutoWS employs a greedy algorithm that iteratively identifies the "bottleneck" layer in the pipeline. It then optimizes compute factors or migrates weights off-chip to satisfy memory constraints while maximizing overall throughput.
  3. Deterministic DMA Scheduler: To prevent hardware engines from competing for limited off-chip bandwidth, AutoWS utilizes a specialized DMA scheduler. It implements a "write burst balancing" strategy, synchronizing the number of data bursts across layers to prevent pipeline stalls and ensure smooth data flow.
AutoWS memory fragmentation strategy and hardware implementation for hybrid weight storage.

Evaluation and Datasets

The framework was evaluated using standard industry benchmarks across various tasks:

  • ImageNet: Classification performance was tested on MobileNetV2, ResNet18, and ResNet50.
  • COCO: Object detection tasks were evaluated using a quantized YOLOv5n model.

Key Insights

  • Throughput Scaling: The authors utilize a "slow-down" factor ($s_l$) to quantify the throughput ratio between the slowest engine and others. This allows the system to scale down bandwidth requirements for faster layers without impacting the total pipeline speed.
  • Optimal Hybridization: The most efficient designs are rarely purely on-chip or purely streaming. By maintaining a hybrid storage model, AutoWS finds a "sweet spot" that respects hardware limits while maintaining high performance.
  • Burst Synchronization: A critical finding was that mismatched data bursts between layers cause significant pipeline stalls. By enforcing uniform burst counts across layers, AutoWS significantly improves pipeline stability.

Key Results

  • BRAM Efficiency: In a ResNet18 implementation on a ZCU102 FPGA, AutoWS achieved a 70% reduction in BRAM usage compared to vanilla pipelined designs with negligible throughput loss.
  • Latency Improvements: For ResNet50 on a U50 FPGA, AutoWS reduced latency from 15.0ms (vanilla) to 3.4ms.
  • Competitive Performance: In object detection (YOLOv5n on ZCU102), AutoWS achieved 8.7ms latency—a 36% improvement over Xilinx’s Vitis AI sequential implementation.
  • Enhanced Scalability: AutoWS successfully deployed ResNet50 on devices where standard pipelined methods were previously infeasible due to memory constraints.

Limitations and Future Work

On extremely resource-constrained devices like the Zedboard, off-chip bandwidth can become the primary bottleneck. In such cases, the overhead of pipelined buffers may result in sequential execution achieving lower latency. Future research will explore combining AutoWS with weight compression and pruning techniques to further mitigate bandwidth limitations. Additionally, the authors plan to integrate this methodology into existing toolflows such as FINN and hls4ml.

Summary updated

Abstract

With the great success of Deep Neural Networks (DNN), the design of efficient hardware accelerators has triggered wide interest in the research community. Existing research explores two architectural strategies: sequential layer execution and layer-wise pipelining. While the former supports a wider range of models, the latter is favoured for its enhanced customization and efficiency. A challenge for the layer-wise pipelining architecture is its substantial demand for the on-chip memory for weights storage, impeding the deployment of large-scale networks on resource-constrained devices. This paper introduces AutoWS, a pioneering memory management methodology that exploits both on-chip and off-chip memory to optimize weight storage within a layer-wise pipelining architecture, taking advantage of its static schedule. Through a comprehensive investigation on both the hardware design and the Design Space Exploration, our methodology is fully automated and enables the deployment of large-scale DNN models on resource-constrained devices, which was not possible in existing works that target layer-wise pipelining architectures. AutoWS is open-source: https://github.com/Yu-Zhewen/AutoWS
BibTeX
@misc{yu2023autows,
  author = {Zhewen Yu and Christos-Savvas Bouganis},
  title = {{AutoWS: Automate Weights Streaming in Layer-wise Pipelined DNN Accelerators}},
  year = {2023},
  eprint = {2311.04764},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.04764},
}
Figure