Abstract
Neural networks achieve state-of-the-art performance in image classification, speech recognition, scientific analysis and many more application areas. Due to the high computational complexity and memory footprint of neural networks, various compression techniques, such as pruning and quantization, have been proposed in literature. Pruning sparsifies a neural network, reducing the number of multiplications and memory. However, pruning often fails to capture properties of the underlying hardware, causing unstructured sparsity and load-balance inefficiency, thus bottlenecking resource improvements. We propose a hardware-centric formulation of pruning, by formulating it as a knapsack problem with resource-aware tensor structures. Evaluated on a range of tasks, including sub-microsecond particle classification at CERN's Large Hadron Collider and fast image classification, the proposed method achieves reductions ranging between 55% and 92% in the DSP utilization and up to 81% in BRAM utilization.
Introduction
Neural networks are notoriously power-hungry and computationally expensive, making them a challenge to deploy on edge devices like FPGAs. While "pruning"—the act of removing unnecessary weights—is a standard technique to slim down these models, there is a catch: standard pruning often creates "unstructured" sparsity. To an FPGA, a scattering of random zeros is often just as expensive as a full matrix because the underlying hardware (DSPs and BRAM) isn't designed to skip over those tiny, isolated gaps.
In their paper, Benjamin Ramhorst and his colleagues from Imperial College London and MIT propose a way to prune networks so that the hardware actually feels the benefits. They identify that for pruning to be effective on FPGAs, we must prune entire groups of weights that share the same hardware resources.
Methodology: Resource-Aware Structured Pruning
The core innovation is a resource-aware structured pruning algorithm integrated into the popular hls4ml library. This approach moves beyond simple magnitude-based pruning by considering how weights are physically mapped to FPGA resources.
Resource-Aware Vectors and the Reuse Factor
The authors leverage the "Reuse Factor" (RF)—a setting in hls4ml that determines how many times a single multiplier is reused for different weights. They group weights into "resource-aware vectors" based on this mapping. If an entire vector is pruned, a physical DSP block or a chunk of BRAM can be removed from the final implementation.
The Knapsack Formulation
Instead of just pruning the smallest weights, the authors treat pruning as a 0-1 knapsack problem. They aim to maximize the "importance" (normalized magnitude) of the weights they keep, while staying under a strict "budget" of hardware resources (DSPs and BRAM). This ensures that every weight removed contributes directly to a reduction in physical resource usage.
Iterative Training and Regularization
The process is iterative. The authors use a resource-aware regularization loss during training to "push" weights that share hardware toward zero together. They then iteratively reduce the resource budget and retrain to recover any lost accuracy, allowing the optimizer to consolidate the network's knowledge into fewer hardware units.
Experimental Evaluation
The researchers evaluated their method on three distinct tasks: High-Energy Jet Tagging (a sub-microsecond particle classification task from CERN’s Large Hadron Collider), SVHN (Street View House Numbers), and Fashion MNIST.
Key Results
The results demonstrate massive hardware savings with minimal impact on accuracy. In some cases, the pruned models actually performed better than the baselines, likely due to the regularizing effect of the pruning process preventing overfitting.
Discussion and Future Work
The methodology highlights that structured sparsity is more valuable than high sparsity for real-time deployment. A model with 70% "structured" sparsity (where whole hardware units are freed) is much more efficient than a model with 90% "unstructured" sparsity.
However, some limitations remain. For the Fashion MNIST model, the authors manually tuned the Reuse Factor for each layer. Automating this "hyper-hardware-parameter" search is a logical next step. Additionally, while the current strategy focuses on DSPs and BRAM, extending it to target Look-Up Table (LUT) heavy architectures for low-precision multiplications could yield further gains.
The methodology is open-sourced and available on GitHub, providing a practical tool for engineers to optimize high-performance AI for FPGA deployment.
Summary updated