Abstract
Modern Deep Neural Networks (DNNs) exhibit varying degrees of sparsity, yet existing hardware accelerators often struggle to balance flexibility with efficiency. Structured sparse accelerators offer high efficiency but are restricted to specific patterns, while unstructured accelerators handle arbitrary sparsity at the cost of a significant "sparsity tax." This paper introduces HighLight, a framework and accelerator architecture based on Hierarchical Structured Sparsity (HSS). By composing multiple simple, hardware-friendly patterns, HighLight achieves the flexibility of unstructured engines with the extreme efficiency of structured ones. Experimental results demonstrate that HighLight delivers a geomean improvement of 6.4× in energy-delay product (EDP) over dense baselines while maintaining a low area overhead.
Introduction
As Deep Neural Networks (DNNs) grow in complexity, exploiting sparsity—the presence of zero-valued weights and activations—has become a primary strategy for improving computational efficiency. However, current hardware support for sparsity is bifurcated. Structured sparse accelerators, such as NVIDIA’s Ampere Sparse Tensor Cores, are optimized for specific N:M patterns (e.g., 2:4), making them highly efficient but unable to exploit higher sparsity levels or adapt to models that do not fit these rigid constraints. Conversely, unstructured sparse accelerators can handle any distribution of zeros but suffer from a "sparsity tax"—complex logic required for non-zero indexing and workload balancing that often offsets the gains from skipping zero computations.
HighLight addresses this gap by introducing Hierarchical Structured Sparsity (HSS), a modular approach that represents diverse sparsity levels by layering simple, structured patterns. This allows for a flexible yet hardware-efficient acceleration framework.
Hierarchical Structured Sparsity (HSS)
The core innovation of HighLight is the decomposition of complex sparsity requirements into a hierarchy of simple "G:H" patterns. In a G:H pattern, at most G elements are non-zero within a block of H elements.
Composition and Modularity
By treating sparsity as a product of fractions, HSS can represent a wide variety of densities. For example, applying a 2:4 pattern at one level and a 3:4 pattern at another level results in a composite density of 37.5%. This modularity allows the hardware to remain "structured" (and thus efficient) while providing the software with the flexibility to target various accuracy-performance trade-offs. To formally define these patterns across different tensor dimensions (such as channels or kernels), the authors utilize a Fibertree abstraction, which precisely maps how hierarchical patterns are applied across ranks.
Reducing the Sparsity Tax
A critical advantage of HSS is the reduction of "Hmax"—the maximum jump or offset the hardware must handle at any single level. By splitting a high sparsity requirement into two hierarchical levels, the complexity and area of the multiplexer logic are reduced linearly. This design choice minimizes the hardware overhead typically associated with flexible sparse processing.
The HighLight Accelerator Architecture
The HighLight accelerator is designed to natively support HSS through specialized hardware components that prioritize data movement efficiency and processing element (PE) utilization.
- Modularized Skipping: The architecture features Sparse Acceleration Features (SAFs) at multiple levels. A "Rank1" SAF manages coarse-grained blocks, while a "Rank0" SAF handles fine-grained elements, mirroring the HSS hierarchy.
- Variable Fetch Management Unit (VFMU): To avoid the high power consumption of complex "intersection" logic found in unstructured accelerators, the VFMU uses a small buffer and shifting logic. This allows the system to stream only necessary non-zero data to the PEs with minimal control overhead.
- Dual-Sided Support: While weights are pruned using HSS patterns, the accelerator also accommodates unstructured sparsity in activations. This is achieved through power-saving "gating," where compute units are momentarily disabled when a zero-valued activation is detected, further improving energy efficiency.
Evaluation
Methodology
The authors evaluated HighLight using a mix of synthetic workloads and representative DNNs:
- Workloads: ResNet50 (CNN), DeiT-small (Vision Transformer), and Transformer-Big (NLP).
- Baselines: Comparisons were made against dense accelerators (TC-like), structured sparse designs (STC-like, S2TA), and unstructured sparse designs (DSTC-like).
- Metrics: Energy-Delay Product (EDP), area overhead, and accuracy-vs-performance Pareto frontiers.
Key Results
- Efficiency Gains: HighLight achieved a geomean of 6.4× better EDP (reaching up to 20.4× in specific cases) compared to dense accelerators. It also outperformed existing sparse baselines by a geomean of 2.7×.
- Pareto Optimality: In accuracy-vs-performance evaluations, HighLight consistently occupied the Pareto frontier, providing the highest hardware speedup for any given level of model accuracy across different pruning regimes.
- Hardware Overhead: The specialized logic required to handle HSS (the SAFs) accounts for only 5.7% of the total chip area, demonstrating that flexibility can be achieved without significant silicon costs.
- Workload Balancing: Because HSS patterns are structured, the hardware can guarantee perfectly balanced workloads across PEs, ensuring maximum hardware utilization and preventing the "tail effect" where some units sit idle while others finish computations.
Conclusion and Future Work
HighLight demonstrates that hierarchical composition is a powerful tool for bridging the gap between rigid structured sparsity and inefficient unstructured sparsity. By leveraging HSS, the HighLight accelerator provides a scalable and efficient solution for modern DNN workloads.
Future research directions include the development of "Dual-Side HSS" pruning algorithms to enforce hierarchical patterns on activations during runtime, as well as further optimizing HSS-aware compilers to automate the mapping of diverse models onto the HighLight architecture. Currently, the framework relies on tools like Condensa for pruning, but tighter integration between the software stack and the hierarchical hardware logic remains a promising area for further exploration.
Summary updated