Abstract
AutoWS is an automated framework designed to enable weight streaming for layer-wise pipelined Deep Neural Network (DNN) accelerators on FPGAs. By intelligently partitioning weights between on-chip and off-chip memory, AutoWS allows large-scale neural networks to be deployed on resource-constrained devices that previously lacked the capacity to house them. Experimental results demonstrate that AutoWS can significantly reduce on-chip memory usage and latency, often outperforming traditional sequential execution models and industry-standard implementations.
Introduction: The Memory Wall in FPGA Acceleration
In FPGA-based DNN acceleration, designers typically choose between two primary architectures: sequential execution, where a single engine processes layers one by one, and layer-wise pipelining, where each layer is assigned a dedicated hardware engine.
While pipelining offers high efficiency and customization, it faces a significant "memory wall." Conventional pipelined designs attempt to store all model weights in the FPGA's on-chip Block RAM (BRAM). For large-scale models like ResNet50, the weight requirements exceed the BRAM capacity of most mid-range FPGAs. This limitation often forces a retreat to sequential execution, which suffers from high latency due to frequent off-chip data movement. AutoWS bridges this gap by enabling pipelined architectures to stream weights dynamically from off-chip DRAM.
Methodology
AutoWS introduces a three-pronged approach to automate weight streaming:
- Hybrid Weight Storage: Rather than an "all-or-nothing" approach, AutoWS partitions weight storage into static regions (permanently on-chip) and dynamic regions (reloaded from off-chip DRAM as needed). This configuration is optimized on a per-layer basis to balance BRAM usage and bandwidth requirements.
- Greedy Design Space Exploration (DSE): Determining the optimal weight partition is a complex optimization problem. AutoWS employs a greedy algorithm that iteratively identifies the "bottleneck" layer in the pipeline. It then optimizes compute factors or migrates weights off-chip to satisfy memory constraints while maximizing overall throughput.
- Deterministic DMA Scheduler: To prevent hardware engines from competing for limited off-chip bandwidth, AutoWS utilizes a specialized DMA scheduler. It implements a "write burst balancing" strategy, synchronizing the number of data bursts across layers to prevent pipeline stalls and ensure smooth data flow.
Evaluation and Datasets
The framework was evaluated using standard industry benchmarks across various tasks:
- ImageNet: Classification performance was tested on MobileNetV2, ResNet18, and ResNet50.
- COCO: Object detection tasks were evaluated using a quantized YOLOv5n model.
Key Insights
- Throughput Scaling: The authors utilize a "slow-down" factor ($s_l$) to quantify the throughput ratio between the slowest engine and others. This allows the system to scale down bandwidth requirements for faster layers without impacting the total pipeline speed.
- Optimal Hybridization: The most efficient designs are rarely purely on-chip or purely streaming. By maintaining a hybrid storage model, AutoWS finds a "sweet spot" that respects hardware limits while maintaining high performance.
- Burst Synchronization: A critical finding was that mismatched data bursts between layers cause significant pipeline stalls. By enforcing uniform burst counts across layers, AutoWS significantly improves pipeline stability.
Key Results
- BRAM Efficiency: In a ResNet18 implementation on a ZCU102 FPGA, AutoWS achieved a 70% reduction in BRAM usage compared to vanilla pipelined designs with negligible throughput loss.
- Latency Improvements: For ResNet50 on a U50 FPGA, AutoWS reduced latency from 15.0ms (vanilla) to 3.4ms.
- Competitive Performance: In object detection (YOLOv5n on ZCU102), AutoWS achieved 8.7ms latency—a 36% improvement over Xilinx’s Vitis AI sequential implementation.
- Enhanced Scalability: AutoWS successfully deployed ResNet50 on devices where standard pipelined methods were previously infeasible due to memory constraints.
Limitations and Future Work
On extremely resource-constrained devices like the Zedboard, off-chip bandwidth can become the primary bottleneck. In such cases, the overhead of pipelined buffers may result in sequential execution achieving lower latency. Future research will explore combining AutoWS with weight compression and pruning techniques to further mitigate bandwidth limitations. Additionally, the authors plan to integrate this methodology into existing toolflows such as FINN and hls4ml.
Summary updated