Deep learning models spend a massive amount of time and energy computing a simple algebraic truth: $0 \times X = 0$. In modern neural networks, zero values are everywhere. They appear statically when weights are pruned to compress a model, and they emerge dynamically at runtime when activation functions like ReLU clamp negative values to zero, when neurons fail to fire in spiking networks, or when attention heads are masked.
In a conventional dense hardware architecture, these zeroes are processed just like any other number. The hardware wastes clock cycles and electrical power loading zeroes from memory and passing them through Multiply-Accumulate (MAC) units. To eliminate this inefficiency, a prominent research direction has emerged: hardware-accelerated computation skipping. By designing specialized architectures that dynamically detect zero-valued operands on both sides of an operation—weights and activations—hardware can bypass redundant calculations entirely, translating mathematical sparsity into direct gains in throughput and energy efficiency.
The Compounding Effect of Dual-Side Sparsity
Why target both weights and activations simultaneously? The answer lies in the compounding nature of mathematical multiplication. If an accelerator only skips computations when a weight is zero (static sparsity), it still wastes energy when a non-zero weight is multiplied by a zero-valued activation.
As shown by Han et al. in their work on the EIE: Efficient Inference Engine on Compressed Deep Neural Network, simultaneously exploiting static weight pruning and dynamic activation sparsity allows hardware to process only the non-zero intersections of data. This dual-side exploitation creates a synergy where efficiency gains multiply.
This effect is further amplified when quantization is introduced. Moons et al. demonstrate in Energy-Efficient ConvNets Through Approximate Computing that scaling down bit-precision maps small, ineffectual values directly to zero. This quantization-induced sparsity complements the natural sparsity of activation functions, giving hardware flags more opportunities to detect zero operands and bypass multiplication cycles.
To systematically evaluate these interactions, Yang et al. introduced the CIMinus: Empowering Sparse DNN Workloads Modeling and Exploration on SRAM-Based CIM Architectures framework, proving that skipping zero-valued bits across both weights and activations provides a compounding efficiency boost even within the rigid constraints of Compute-In-Memory (CIM) architectures.
Identifying Zeroes: Masking and Detection
Before hardware can skip a computation, it must identify that an operand is zero with less overhead than the computation itself would consume. This is achieved through hardware-level masking and look-ahead detection modules.
[Input Activation Mask] ---> \
( Bitwise AND ) ---> [Compute Decision]
[Synaptic Weight Mask] ---> / |
v
(Execute MAC or Skip Cycle)
In Spiking Neural Networks (SNNs), where sparsity is exceptionally high due to sporadic neuron firing, Li et al. developed a bitwise approach in FireFly-S: Exploiting Dual-Side Sparsity for Spiking Neural Networks Acceleration with Reconfigurable Spatial Architecture. The architecture performs a rapid bitwise AND operation between input spike masks and weight masks. If either side is zero, the system bypasses the execution logic entirely.
For workloads with dynamic runtime variations like Transformers, Tuli et al. utilize specialized sparsity-aware modules within a custom ASIC architecture in AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference With Transformers, employing binary masks to filter out dynamic activation zeroes on-the-fly.
Alternatively, the detection can happen during data ingestion. In High Area/Energy Efficiency RRAM CNN Accelerator with Pattern-Pruning-Based Weight Mapping Scheme, Yu et al. deploy an input preprocessing unit featuring all-zero detection modules. This unit filters incoming activations based on known non-zero weight locations, ensuring that the downstream crossbars never waste dynamic energy on zero-value operations.
Mechanisms of Skipping: Gating vs. Scheduling
Once a zero is detected, the hardware must execute the "skip." Accelerators generally handle this via two distinct mechanisms: power gating (which saves energy) and dynamic scheduling (which saves both energy and time).
Power and Clock Gating
In this paradigm, the execution timeline remains fixed, but the hardware prevents internal signals from toggling. As Zhu et al. explain in An Efficient Hardware Accelerator for Structured Sparse Convolutional Neural Networks on FPGAs, skipping cycles for zero-valued weights directly accelerates throughput, while gating logic for zero-valued activations cuts down power consumption by freezing the hardware switching states. This ensures that even if a compute unit idles for a cycle, its dynamic power draw drops to near zero.
Dynamic Scheduling and Routing
To save time and maximize throughput, more aggressive architectures compress the execution schedule by routing around zeroes so that ALUs are never left idling. Mahmoud et al. implement this in TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training using a hardware scheduler paired with a low-cost multiplexer interconnect. The scheduler ingests bit vectors representing zero values from staging buffers, pairs up only the effectual (non-zero) operands, and dynamically routes them to active multipliers.
Similarly, for Graph Convolutional Networks (GCNs), Sohrabizadeh et al. design an on-the-fly hardware pruner and arbiter in SPA-GCN: Efficient and Flexible GCN Accelerator with Application for Graph Similarity Computation. This system dynamically skips zero elements during feature transformation, inserting pipeline bubbles only when strict data dependencies demand them, thereby maximizing hardware utilization.
Balancing Structure and Flexibility
Unstructured sparsity offers the highest theoretical compression, but it creates irregular, unpredictable data streams that cause memory bottlenecks. To resolve this, researchers use hardware-software co-design to enforce structured or hierarchical patterns that the hardware can exploit cleanly.
Sun et al. address this at the block level in 3D CNN Acceleration on FPGA using Hardware-Aware Pruning. By pruning weights in structured blocks that match the physical tiling of an FPGA convolution engine, they integrate a simple "block enable" signal. If an entire block is marked as pruned, the signal prevents it from being loaded into memory or processed, translating structural model sparsity directly into computational savings.
To achieve both the predictability of structured execution and the high compression of unstructured patterns, Wu et al. proposed Hierarchical Structured Sparsity (HSS) in HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity. Their accelerator keeps model weights in a rigid, hierarchical structure for clean hardware scheduling, while handling unpredictable, unstructured activation sparsity via dynamic power gating.
[Hierarchical Weight Patterns] ---> Predictable Memory Scheduling \
===> High Efficiency
[Unstructured Activations] ---> Dynamic Power Gating /
This hardware-software synergy is not limited to custom ASICs or FPGAs. Raghunathan et al. demonstrated in SparCE: Sparsity Aware General-Purpose Core Extensions to Accelerate Deep Neural Networks that general-purpose processors (GPPs) can exploit dynamic sparsity. By adding micro-architectural and Instruction Set Architecture (ISA) extensions, software can explicitly flag sparse data regions, allowing a general-purpose core to skip redundant computations without incurring prohibitive hardware overhead.
Generalization Across Diverse Workloads
The core intuition of dual-side computation skipping extends well beyond standard convolutional and transformer inference:
- Multi-Task Inference: In MIME: Adapting a Single Neural Network for Multi-task Inference with Memory-efficient Dynamic Pruning, Bhattacharjee et al. combine dynamic activation pruning with an output-stationary systolic array. This allows the hardware to skip MAC operations based on input-dependent task contexts on-the-fly, eliminating the need to store multiple, permanently compressed models for different tasks.
- Non-Deep Learning Graph Algorithms: The same principles apply to robotics and spatial computing. In BLITZCRANK: Factor Graph Accelerator for Motion Planning, Yu et al. design an accelerator that omits zero-value computations within hardware matrix decomposition blocks. By exploiting the mathematical symmetries and chain-like sparsity inherent to factor graphs in localization algorithms, the architecture significantly reduces physical FPGA resource utilization while maintaining full operational accuracy.
Ultimately, whether applied to general-purpose cores, specialized systolic arrays, or compute-in-memory crossbars, the objective remains identical: by engineering hardware to actively detect and skip the mathematical void of zeroes, architectures achieve massive breakthroughs in speed and power efficiency.
Key Foundational Work
Energy-Efficient ConvNets Through Approximate Computing (2016) — contributed 2 concept elements to this direction.
EIE: Efficient Inference Engine on Compressed Deep Neural Network (2016) — contributed 1 concept elements to this direction.
SparCE: Sparsity Aware General-Purpose Core Extensions to Accelerate Deep Neural Networks (2019) — contributed 1 concept elements to this direction.