Modern deep learning has largely moved away from the simple Rectified Linear Unit (ReLU) in favor of "smoother" activation functions like SiLU or GELU. While these functions improve gradient flow and convergence, they suffer from a major efficiency drawback: they are dense. In a dense model, nearly every neuron fires for every input, forcing the hardware to perform massive amounts of computation and memory transfers.
A growing body of research suggests we can reclaim this lost efficiency by intentionally inducing activation sparsity—forcing a large percentage of neurons to output exactly zero. The most effective mechanism for this is threshold shifting, where the activation function is modified to ignore weak signals, allowing hardware to skip the associated computations entirely.
The Mechanism: Shifting the Threshold
The intuition behind threshold shifting is that not all neural signals are equally important. In a standard ReLU, any value greater than zero is passed through. By applying a positive bias shift, $f(x) = \max(0, x - b)$, we can "raise the bar" for what constitutes a meaningful signal.
Iman Mirzadeh, Mohammad Rastegari, and colleagues demonstrate in ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models that shifting the threshold allows for tunable control over the trade-off between accuracy and sparsity. By increasing the bias $b$, practitioners can force over 90% of neurons to remain inactive.
This concept extends beyond simple ReLUs. Zhengyan Zhang and Xu Han found that the ReLU² function—a squared ReLU—is particularly effective for Large Language Models (LLMs), providing high sparsity with minimal performance degradation and a massive reduction in I/O overhead ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs.
ReLUfication: Converting Dense Models to Sparse
We don't always need to train a sparse model from scratch. ReLUfication is the process of swapping dense activations (like SiLU) in a pre-trained model for sparse ones.
To prevent the model from collapsing after this swap, researchers use a short phase of "recovery fine-tuning." Chenyang Song and the team behind ProSparse use progressive $L_1$ regularization alongside threshold shifting to achieve ~89% sparsity in LLaMA2 models. This transformation enables up to a 4.5x inference speedup on standard hardware by pruning weak signals that contribute little to the final output.
Why Sparsity Enables Speed
Activation sparsity is not just a mathematical curiosity; it is a hardware accelerator. When a neuron's activation is zero, the hardware can skip the corresponding weight multiplications in the next layer.
- Memory Efficiency: In LLM in a flash, Keivan Alizadeh and others show that sparsity allows models to run on devices with limited RAM. By only loading the weights for "active" neurons from flash memory just-in-time, they achieved 25x faster inference.
- Temporal Sparsity: In recurrent networks, data often changes slowly. Shih-Chii Liu and Daniel Neil introduced Delta Networks, which only update neurons whose activations change significantly over time Delta Networks for Optimized Recurrent Network Computation.
- Iterative Convergence: In Deep Equilibrium (DEQ) models, Zuowen Wang and Niklas Hahn use thresholding to skip updates for dimensions that have already stabilized, reducing operations by over 70% DeltaDEQ: Exploiting Heterogeneous Convergence for Accelerating Deep Equilibrium Iterations.
Theoretical Foundations: Sublinear Training
Beyond inference, threshold shifting offers a path to faster training. Standard training costs are quadratic relative to model width. However, Zhao Song and colleagues have proven that by using shifted ReLUs to induce extreme sparsity at initialization, one can achieve sublinear training costs per iteration Does Preprocessing Help Training Over-parameterized Neural Networks?.
By ensuring only a small fraction of neurons fire for any given input, the updates can be restricted to a sublinear subset of the network, providing a theoretical blueprint for training massive models with significantly fewer resources Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time.
Beyond LLMs: Retrieval and Perception
The principle of thresholding for sparsity is versatile:
- Search: Tiancheng Zhao uses a trainable bias to threshold interaction scores in SPARTA, creating sparse representations that can be indexed and queried as efficiently as traditional keyword searches.
- 3D Reconstruction: Weixing Zhang uses spiking neurons with learnable thresholds to prune low-opacity noise in Gaussian Splatting, leading to cleaner surface reconstructions with fewer parameters Spiking GS.
- Neuromorphic Computing: Yingfu Xu and Amirreza Yousefzadeh demonstrate that optimizing thresholds via surrogate gradients allows Artificial Neural Networks to match the extreme energy efficiency of Spiking Neural Networks Event-based Optical Flow on Neuromorphic Processor.
By shifting the threshold, researchers are moving away from the "always-on" nature of traditional deep learning toward a more biological, "event-driven" paradigm where computation only happens when it is truly necessary.