Exploiting Activation Sparsity Beyond ReLU

24 concepts23 papers1 proposal

In deep learning, the most efficient computation is the one you never have to perform. For years, the field relied on a simple mathematical quirk to achieve this efficiency: the Rectified Linear Unit (ReLU) activation function, defined as $f(x) = \max(0, x)$. Because ReLU outputs an exact zero for any negative input, deep neural networks naturally exhibited high activation sparsity—a phenomenon where a massive percentage of a network's neurons remain completely silent for any given input token or image patch.

When an activation is exactly zero, hardware accelerators can completely skip loading the corresponding weights from memory and bypass the associated multiply-accumulate (MAC) operations. This "hard sparsity" formed the backbone of both hardware and algorithmic efficiency gains.

However, modern deep learning has largely abandoned ReLU. To improve gradient flow and model convergence, state-of-the-art architectures—particularly Large Language Models (LLMs) like LLaMA and Mistral—have transitioned to smooth, continuous activation functions such as GELU, SiLU, and SwiGLU. While these functions boost accuracy, they introduce a major problem: they are fundamentally non-sparse, producing non-zero outputs for nearly every input.

This shift has created the "ReLU Barrier." Traditional hardware accelerators, pruning algorithms, and dynamic routing mechanisms break down entirely when applied to these modern, smooth activation functions. A rapidly growing research direction focuses on reclaiming this lost efficiency by inducing, predicting, or approximating activation sparsity in modern, non-sparse architectures.


The Golden Age of Hard Sparsity

To understand why the shift to smooth activations was so disruptive, we must first look at how deeply the ecosystem was optimized around ReLU's exact zeros.

On the hardware side, engineers built specialized systems to exploit zero-valued data. For instance, Minsoo Rhu et al. developed a compressing DMA engine (Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks) that compresses activation data on-the-fly to reduce PCIe bottlenecking during training. Similarly, Brandon Reagen et al. designed MASR (MASR: A Modular Accelerator for Sparse RNNs), a hardware accelerator that directly bypasses redundant calculations when activations drop to zero. Both systems achieved massive energy and speed gains, but their core compression and skipping mechanisms were explicitly contingent on the high volume of exact zeros produced by ReLU.

On the algorithmic side, researchers leveraged this natural inactivity to shrink models. Yu-Hsun Lin et al. introduced Macroblock Scaling (MBS: Macroblock Scaling for CNN Model Reduction), which measures "effective FLOPs"—computations resulting in non-zero data—to identify and prune redundant channels. Yuchu Fang et al. utilized the "dying ReLU" phenomenon in FIREPruning (FIREPruning: Learning-based Filter Pruning for Convolutional Neural Network Compression) to predict and eliminate inactive filters based on weight statistics. Even structural importance metrics, such as the batch-normalization-based approach proposed by Junghun Oh et al. (Batch Normalization Tells You Which Filter is Important), were optimized under the assumption of zero-centered, sparse feature distributions.

Perhaps the most striking realization of natural sparsity came from Maosong Sun et al. with their work on MoEfication (MoEfication: Transformer Feed-forward Layers are Mixtures of Experts). They discovered that in ReLU-based Transformers, the Feed-Forward Networks (FFNs) are naturally sparse; only a tiny fraction of neurons "fire" for any given token. By grouping these naturally co-activating neurons into a Mixture-of-Experts (MoE) structure post-hoc, they could skip 70-90% of the computation with negligible performance loss.


The Smooth Activation Tax

The transition to modern architectures broke these paradigms. As Iman Mirzadeh details in ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models, functions like SiLU and GELU are non-sparse by design. They trade away the hard zero-threshold to maintain smooth gradients for negative values, which aids optimization but forces the underlying hardware to load every single weight and execute every calculation.

ReLU (Sparse):   [ -3.4,  1.2, -0.1,  5.5 ]  ==>  [  0.0,  1.2,  0.0,  5.5 ]  (50% Sparse)
SiLU (Dense):   [ -3.4,  1.2, -0.1,  5.5 ]  ==>  [-0.11,  0.92, -0.05,  5.47]  (0% Sparse)

As a result, post-hoc conversion techniques fail. Jaeseong Lee et al. (Breaking ReLU Barrier: Generalized MoEfication for Dense Pretrained Models) and Filip Szatkowski et al. (Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion) both note that because smooth activations lack a distinct inactive region, traditional sparse routing mechanisms have no clear signal for skipping computations.

This density even alters security profiles. Jona te Lintelo et al. demonstrated this in The SkipSponge attack: Sponge weight poisoning of deep neural networks, showing that energy-wasting "sponge attacks"—which manipulate biases to minimize sparsity and maximize hardware energy consumption—struggle to generalize to smooth activations because the baseline models are already computationally dense.

To bridge this gap, researchers are developing four primary strategies to re-introduce activation sparsity into smooth neural networks.


Strategy 1: Post-Hoc Thresholding and Selective Sparsification

The most direct way to reclaim sparsity is to forcefully zero out low-magnitude activations during inference. If an activation's value is close to zero, its contribution to the final matrix multiplication is likely negligible.

The authors of Training-Free Activation Sparsity in Large Language Models (TEAL) show that simply applying a magnitude-based threshold to zero out low-value activations in modern models like Llama-3 and Mistral can unlock up to a 1.8× wall-clock speedup without requiring any model retraining.

However, naive uniform thresholding can severely degrade a model's accuracy because different channels exhibit different sensitivities. To address this, Junhui He et al. proposed CHESS (CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification). Instead of a global threshold, CHESS computes channel-specific thresholds and pairs them with custom sparse kernels, allowing the system to skip inactive neurons in SwiGLU and GeGLU layers while preserving accuracy.


Strategy 2: Low-Rank Decomposition for "Soft" Sparsity

Rather than forcing a hard zero boundary on smooth functions, an alternative approach embraces the concept of "soft" sparsity by combining activation distributions with matrix factorization.

Even though smooth activations do not produce exact zeros, their magnitude distributions are highly skewed. X. T. Huang et al. leverage this in SoLA (SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression). They observe that a small subset of critical "Prime Neurons" dominate the network's outputs, while the vast majority act as "Marginal Neurons" with low-magnitude, low-variance outputs. Instead of pruning the marginal neurons entirely, SoLA applies Singular Value Decomposition (SVD) to compress the weight spaces associated with them, maintaining high fidelity at a fraction of the computational cost.

[ Smooth Activation Vector ] 
       │
       ├─► High-Magnitude "Prime Neurons"    ──► Compute with Full Weight Matrix
       │
       └─► Low-Magnitude "Marginal Neurons"  ──► Compute with Compressed Low-Rank (SVD) Matrix

Similarly, Zhenyu Zhang et al. introduced R-Sparse (R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference), which pairs input channel selection with low-rank weight approximations. This allows the system to bypass the lack of natural zero values in SiLU or GELU layers without requiring expensive retraining or active channel predictors.


Strategy 3: Predictive Routing and Offloading

If we cannot rely on the activation function itself to produce zeros, can we predict which neurons will be unimportant before we even compute the activation layer?

Guang Yan et al. applied this predictive intuition to secure computing in Comet (Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity). In Secure Multi-Party Computation (MPC), evaluating non-linear activations like GeLU or SwiGLU is incredibly expensive due to communication overhead. Comet trains a lightweight, hardware-friendly predictor to forecast which neurons will have negligible outputs, allowing the system to skip both the cryptographic computation and the associated communication rounds.

This predictive approach also enables novel hardware offloading configurations. Lian Liu et al. developed Hermes (Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMM), a system that keeps highly active "hot" neurons on the GPU while offloading "cold" neurons to specialized memory modules (NDP-DIMMs). While they note that non-sparse functions like SiLU require an extra conversion step to map cleanly to this architecture, predicting and isolating these inactive zones allows consumer-grade hardware to run massive models.


Strategy 4: Inducing Sparsity by Design During Training

The final strategy avoids post-hoc fixes altogether by modifying the training process to force smooth architectures to behave sparsely.

Haizhong Zheng et al. introduced the Learn-To-be-Efficient (LTE) framework (Learn To be Efficient: Build Structured Sparsity in Large Language Models), which trains standard SwiGLU-based LLMs to dynamically activate only a fraction of their neurons using a custom Mixture-of-Experts (MoE) approach. Instead of relying on natural sparsity, LTE explicitly teaches the model to route tokens to a sparse subset of parameters during training.

From an optimization perspective, Pierluca D'Oro et al. proposed Demon Pruning (Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons). They flipped the traditional view of "dying neurons" from a training failure into an asset. By injecting asymmetric noise and utilizing a specialized regularization schedule, they actively push neurons into flat, gradient-free saturation regions during training, creating highly compressible models out of inherently dense architectures.

Finally, Zhengyan Zhang explored whether these sparse training benefits can be captured at the very beginning of a model's lifecycle (Exploring the Benefit of Activation Sparsity in Pre-training). By empirically validating sparse training across modern Gated Linear Units (GLUs), this work demonstrates that the computational advantages of activation sparsity do not have to be a post-processing afterthought—they can be leveraged to drastically lower the multi-million dollar computational tax of foundational model pre-training.

Key Foundational Work

Joint Regularization on Activations and Weights for Efficient Neural Network Pruning (2019) — contributed 1 concept elements to this direction.
MASR: A Modular Accelerator for Sparse RNNs (2019) — contributed 1 concept elements to this direction.
Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_activation_dependent_sparsity_4140300f,
  title = {activation-dependent sparsity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-dependent-sparsity/rdr_4140300f67c54b83ac5e65d8127d4cab},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_activation_dependent_sparsity_4140300f_v1,
  title = {activation-dependent sparsity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-dependent-sparsity/rdr_4140300f67c54b83ac5e65d8127d4cab/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake