Sparsing Law: Towards Large Language Models with Greater Activation Sparsity

Yuqi Luo, Chenyang Song, Xu Han, Yingfa Chen, Chaojun Xiao, Zhiyuan Liu, Maosong Sun

Presented at ICML 2025, this research investigates activation sparsity—the phenomenon where many neurons in a Large Language Model (LLM) contribute minimally to the final output for a given input. While researchers have long observed that LLMs are "lazy," this work provides a rigorous framework to measure and predict this behavior as models scale.

The authors propose the "Sparsing Law," a set of quantitative observations describing how architecture and training data influence sparsity. Their findings demonstrate how to build models that are over 90% sparse without sacrificing intelligence.

The Problem of Activation Sparsity

In the Feed-Forward Networks (FFNs) of LLMs, many neurons exhibit near-zero activation values for any specific token. Identifying these "weakly-contributing" neurons allows for skipping their computation, potentially leading to significant inference speedups.

Mechanism of activation sparsity in Feed-Forward Networks (FFNs).

However, three major hurdles previously hindered progress:

  1. Measurement Gap: Standard metrics like counting zeros in ReLU do not apply to activation functions like SiLU, which do not produce true zeros.
  2. Influential Factors: The relationship between model size, depth, training data volume, and sparsity was poorly understood.
  3. Design Recipes: There was no established guide for intentionally training highly sparse models from scratch.

Methodology: The CETT-PPL-1% Metric

To address the measurement gap, the researchers developed the CETT-PPL-1% metric. Adapted from the "Cumulative Errors of Tail Truncation" (CETT) algorithm, it identifies a layer-wise threshold for "weak" neurons. The metric uses a binary search to find the point where skipping neurons increases the model's Perplexity (PPL) by exactly 1%. This ensures the metric is performance-centric and generalizable across different activation functions.

The team conducted experiments on models ranging from 0.1B to 1.2B parameters, varying activation functions (ReLU vs. SiLU), training data volume (up to 800B tokens), and width-depth ratios.

Key Insights: The Sparsing Law

The systematic study revealed several critical properties of activation sparsity:

  • ReLU is the Sparsity King: While SiLU is common in models like Llama, ReLU is far superior for sparsity. Crucially, ReLU models follow a logspace power-law, becoming sparser as they are trained on more data. In contrast, SiLU models tend to become denser with more training.
  • Scale Insensitivity: The theoretical limit of sparsity is remarkably consistent across model sizes (e.g., from 0.1B to 1.2B parameters). This suggests that neuron specialization is a fundamental architectural property rather than a byproduct of over-parameterization.
  • Depth Promotes Sparsity: Deeper and narrower models (smaller width-depth ratios) are naturally more sparsely activated, though excessive depth can lead to training instability.
Performance comparison across different sparsity ratios for ReLU and SiLU models.

Results and Performance

By applying the "Sparsing Law" (using ReLU, high data volume, and optimal depth), the authors trained a 2.4B parameter proof-of-concept model that reached 93.52% activation sparsity.

Using PowerInfer, an inference engine designed to exploit sparsity, this model achieved a 4.1× speedup compared to dense implementations in llama.cpp. Remarkably, despite utilizing only ~6.5% of its neurons per token, the sparse ReLU model maintained performance parity with mainstream SiLU-based models on complex benchmarks like MMLU, GSM8K, and HumanEval.

Conclusion and Future Outlook

While the results are promising for inference efficiency, challenges remain. Deeper, sparser models can be less efficient to train due to sequential dependencies. Furthermore, sparsity levels vary by data distribution (e.g., code vs. natural language). Future work will likely focus on hardware-software co-design to better handle the irregular, sparse computations that these models require, moving beyond current block-based GPU optimizations.

Summary updated

BibTeX
@inproceedings{luo2025sparsing,
  author = {Yuqi Luo and Chenyang Song and Xu Han and Yingfa Chen and Chaojun Xiao and Zhiyuan Liu and Maosong Sun},
  title = {{Sparsing Law: Towards Large Language Models with Greater Activation Sparsity}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=SBUc5wirM8},
}
Figure