HASS: Hardware-Aware Sparsity Search for Dataflow DNN Accelerator

Zhewen Yu, Sudarshan Sreeram, Krish Agrawal, Junyi Wu, Alexander Montgomerie-Corcoran, Cheng Zhang, Jianyi Cheng, Christos-Savvas Bouganis, Yiren Zhao

Abstract

Deep Neural Networks (DNNs) require significant computational resources, leading to the adoption of sparsity and dataflow architectures. However, co-optimizing these two strategies is challenging due to the rigid pipelined nature of dataflow accelerators. This paper presents HASS (Hardware-Aware Sparsity Search), a framework that integrates hardware performance metrics directly into the pruning search loop. By co-designing unstructured weight and activation pruning with a custom sparse dataflow architecture, HASS achieves up to a 4.2× efficiency improvement over existing state-of-the-art sparse designs while maintaining high classification accuracy on ImageNet.

Introduction

The increasing complexity of Deep Neural Networks (DNNs) has necessitated the development of specialized hardware and optimization techniques. Sparsity—the practice of pruning non-essential weights and activations—and dataflow architectures—which utilize pipelined processing for high throughput—are two prominent solutions. Despite their individual successes, combining them effectively remains a significant hurdle. Traditional pruning methods are often "hardware-agnostic," focusing on parameter reduction without considering the specific constraints of the target hardware. This paper introduces HASS, a hardware-aware framework designed to bridge the gap between software-level sparsity and hardware-level efficiency in dataflow accelerators.

The Challenge of Dataflow Sparsity

Dataflow architectures are characterized by a deeply pipelined structure where data moves sequentially through hardware units corresponding to different layers. In such systems, the overall throughput is dictated by the slowest layer, or the bottleneck. Hardware-agnostic pruning often fails in this context because reducing non-zero parameters in a non-bottleneck layer does not improve system-wide performance. Furthermore, traditional design flows treat hardware as a post-processing step, missing opportunities to reallocate resources dynamically based on layer-wise sparsity.

Methodology: The HASS Framework

HASS addresses these challenges by implementing a software-hardware co-design loop. The framework moves away from sequential "prune-then-deploy" methodologies in favor of an iterative search that considers real-time hardware metrics.

The HASS hardware-aware co-design loop, integrating pruning search with hardware modeling and optimization.

The HASS framework consists of four primary components:

  1. Unstructured Pruning: Magnitude-based pruning is applied to both static weights and dynamic activations, with unique thresholds determined for each layer to preserve accuracy.
  2. Sparse Dataflow Architecture: A custom Sparse Vector Dot Product Engine (SPE) was developed. It incorporates clipping modules to eliminate near-zero values and a dynamic scheduler with a round-robin arbiter to skip zero-valued computations.
  3. Design Space Exploration (DSE): Utilizing a Tree-structured Parzen Estimator (TPE), HASS searches for optimal pruning thresholds. The objective function includes accuracy, throughput, and DSP utilization.
  4. Rate Balancing: HASS optimizes the pipeline by reallocating hardware resources (such as DSPs) from sparse, fast layers to bottleneck layers, maximizing overall efficiency.

Experimental Results

The HASS framework was evaluated using the ImageNet dataset across several models, including ResNet-18, ResNet-50, and the MobileNetV3 family. The experiments were conducted on an AMD Xilinx Alveo U250 FPGA.

Pareto frontier of Top-1 accuracy vs. operations density, comparing HASS against state-of-the-art sparse accelerators.

The results indicate that HASS significantly advances the Pareto frontier for accuracy and performance. Key findings include:

  • Efficiency Improvements: HASS achieved 1.3× to 4.2× higher operation density (images/cycle/DSP) compared to existing sparse designs like HPIPE and PASS.
  • Throughput Gains: For MobileNetV3, HASS reached a throughput of 4895 images per second. Compared to dense implementations, HASS provided speedups of up to 5.3× for ResNet-50.
  • Accuracy Preservation: These performance gains were achieved with minimal impact on accuracy, typically losing less than 0.6 percentage points relative to previous best-in-class sparse implementations.

Conclusion

HASS demonstrates that integrating hardware awareness into the sparsity search process is essential for maximizing the performance of dataflow accelerators. By co-optimizing pruning and hardware resource allocation, HASS overcomes the limitations of hardware-agnostic methods, providing a scalable and efficient path for deploying sparse DNNs on specialized hardware. Future work may involve exploring fine-tuning steps to further recover accuracy and extending the framework to support more diverse pruning strategies.

Summary updated

Abstract

Deep Neural Networks (DNNs) excel in learning hierarchical representations from raw data, such as images, audio, and text. To compute these DNN models with high performance and energy efficiency, these models are usually deployed onto customized hardware accelerators. Among various accelerator designs, dataflow architecture has shown promising performance due to its layer-pipelined structure and its scalability in data parallelism. Exploiting weights and activations sparsity can further enhance memory storage and computation efficiency. However, existing approaches focus on exploiting sparsity in non-dataflow accelerators, which cannot be applied onto dataflow accelerators because of the large hardware design space introduced. As such, this could miss opportunities to find an optimal combination of sparsity features and hardware designs. In this paper, we propose a novel approach to exploit unstructured weights and activations sparsity for dataflow accelerators, using software and hardware co-optimization. We propose a Hardware-Aware Sparsity Search (HASS) to systematically determine an efficient sparsity solution for dataflow accelerators. Over a set of models, we achieve an efficiency improvement ranging from 1.3$\times$ to 4.2$\times$ compared to existing sparse designs, which are either non-dataflow or non-hardware-aware. Particularly, the throughput of MobileNetV3 can be optimized to 4895 images per second. HASS is open-source: \url{https://github.com/Yu-Zhewen/HASS}
BibTeX
@inproceedings{yu2024hass,
  author = {Zhewen Yu and Sudarshan Sreeram and Krish Agrawal and Junyi Wu and Alexander Montgomerie-Corcoran and Cheng Zhang and Jianyi Cheng and Christos-Savvas Bouganis and Yiren Zhao},
  title = {{HASS: Hardware-Aware Sparsity Search for Dataflow DNN Accelerator}},
  booktitle = {FPL},
  year = {2024},
  url = {https://openreview.net/forum?id=J0B6vNq4f9},
}
Figure