Deep neural networks are notoriously over-parameterized. To deploy these massive models on resource-constrained edge devices, researchers traditionally turn to static pruning—permanently deleting weights, channels, or layers that contribute the least to the model's overall performance. While effective, static pruning suffers from a fundamental limitation: it permanently reduces the model's total capacity. A rare or highly complex input might require a specialized feature channel that was pruned away simply because it was redundant for 99% of common training samples.
Dynamic activation sparsity (also known as input-dependent dynamic pruning) flips this paradigm. Instead of permanently removing weights at compile time, the network adaptively decides which neurons, channels, or layers to execute at runtime, conditioned entirely on the specific input. As noted by Lingao Xiao et al. in their survey on Structured Pruning for Deep Convolutional Neural Networks: A survey, this allows a model to execute a smaller, highly efficient subset of filters for simple inputs while reserving its full computational capacity for difficult cases.
From a theoretical perspective, Pranjal Awasthi et al. demonstrate in Learning Neural Networks with Sparse Activations that this structural design provides significant statistical and computational advantages, allowing models to maintain high representational capacity while reducing the effective complexity of the function class during learning.
How Networks Decide: Input-Dependent Gating
The core challenge of dynamic activation sparsity is determining which pathways to skip without spending more compute on the decision itself than is saved by the pruning. Researchers have converged on a few elegant mechanisms to predict activation relevance on the fly.
1. Lightweight Predictor Modules
The most direct approach embeds tiny, auxiliary gating networks alongside the primary architecture. These modules inspect the intermediate features of an input and generate a sparse mask. For example, Jinting Chen et al. in Self-Adaptive Network Pruning introduce "Saliency-and-Pruning Modules" (SPMs) that predict channel importance per sample.
Similarly, Haoli Bai et al. demonstrate in Dynamically pruning segformer for efficient semantic segmentation that a lightweight predictor can generate input-dependent masks during each forward pass. This allows a network to retain a massive parameter space for versatility while executing a highly sparse, efficient computation path for any given instance.
2. Base vs. Conditional Paths
Rather than using a separate predictor module, a network can use a fraction of its own standard computation to forecast its downstream needs. Weizhe Hua et al. introduce this concept in Channel Gating Neural Networks by partitioning convolutional layers into a primary base path and a secondary conditional path.
Input ───► [ Base Path (Partial Sums) ] ───► Gating Function
│ │
│ (If Salient) │ (Skip if Low Saliency)
▼ ▼
[ Conditional Path ] ────────────────► Skip Remaining Channels
The partial sums computed by the base path serve as a cheap proxy for the full layer's behavior. A gating function evaluates these partial results to dynamically skip the conditional path at specific spatial locations, slashing floating-point operations (FLOPs) with minimal accuracy loss.
3. Contextual Metadata Gating
Decisions do not always have to be extracted from the hidden activations themselves; they can also be driven by external context. In Context-Driven Dynamic Pruning for Large Speech Foundation Models, Masao Someki et al. incorporate external metadata—such as speaker embeddings, acoustic environment features, and linguistic priors—to make layer-specific pruning decisions. This allows the network to adapt its computational load based on the environmental characteristics of the input signal before the deep layers are even processed.
The Granularity of Dynamic Sparsity
Dynamic activation sparsity can be applied at various structural granularities depending on the architecture of the model:
- Channel-Level Selection: In convolutional networks, Yulong Wang et al. show in Dynamic Network Pruning with Interpretable Layerwise Channel Selection that using discrete decision units to select channel masks on the fly can reduce computational costs by over 50% while maintaining high classification accuracy.
- Attention and Feed-Forward Blocks: For Transformer architectures, Zejian Liu et al. develop EBERT: Efficient BERT Inference with Dynamic Structured Pruning, which analyzes the
[CLS]token to dynamically skip specific attention heads and feed-forward channels that are redundant for a given text snippet. - Layer and Block-Level Skipping: In distributed or federated environments, Daoyuan Chen et al. propose Efficient Personalized Federated Learning via Sparse Model-Adaptation, utilizing a lightweight gating mechanism to predict which entire blocks of a shared global model should be activated or bypassed for a given client's input.
The Hardware Execution Bottleneck
If dynamic activation sparsity offers such clear theoretical and computational advantages, why hasn't it completely replaced dense inference? The answer lies in the hardware-software co-design bottleneck.
Traditional hardware accelerators—such as systolic arrays and modern GPUs—are highly optimized for dense, regular, and deterministic matrix operations. As Zhi Gang Liu et al. explain in Systolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inference, while static weight sparsity can be efficiently mapped to hardware at compile time, exploiting dynamic activation sparsity remains a significant challenge due to its unpredictable nature.
Because the locations of zero-valued activations change with every single input—a reality emphasized by Shijie Cao et al. in SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization—standard hardware cannot easily predict memory access patterns or pipeline computations. This unpredictability often leads to underutilized compute units or control overhead that cancels out the theoretical FLOP savings.
To bridge this gap, researchers are actively developing hardware-software co-designs. For instance, Chen Zhang et al. propose a Dual-side Sparse Tensor Core that modifies GPU architectures to use an outer-product computation primitive and a bitmap-based encoding format, allowing hardware to skip both static weight zeros and dynamic activation zeros simultaneously.
Furthermore, Zhewen Yu et al. demonstrate in HASS: Hardware-Aware Sparsity Search for Dataflow DNN Accelerator that accelerators must be designed with dataflow flexibility in mind, ensuring that runtime activation variability can be handled without stalling the execution pipelines that give deep learning hardware its massive speed advantages.
Key Foundational Work
Batched Lazy Decision Trees (2016) — contributed 1 concept elements to this direction.
Elastic Responding Machine for Dialog Generation with Dynamically Mechanism Selecting (2018) — contributed 1 concept elements to this direction.