Training a neural network to be sparse—whether by removing weights, deactivating neurons, or selecting a subset of features—is notoriously difficult to do from scratch. If you initialize a model with high sparsity, it often suffers from "representation collapse," where the model fails to capture the signal and outputs only noise. As Martim Lisboa and Guillaume Bellec observe in Spiking Music: Audio Compression with Event Based Auto-encoders, enforcing strict bit-rate constraints too early prevents the model from learning the underlying structure of the data.
The solution emerging across the field is progressive sparsification: a multi-phase training curriculum that begins with a dense or low-sparsity state and gradually "thins out" the network over time.
The Intuition of the Dense Warm-up
Why does starting dense help? In the early stages of training, a model needs to explore the high-dimensional landscape of the data manifold. A dense model has the "representational capacity" to see all possible correlations. If we prune or mask the model at initialization, we might accidentally kill the very neurons or connections that would have eventually become the most important.
By using a dense warm-up, researchers like Jianfei Chen in ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing allow the model to establish stable feature representations before enforcing selection constraints. This "exploration phase" ensures that when the model is finally forced to be sparse, it chooses its active components based on learned importance rather than random chance.
Scheduling the Transition
The path from dense to sparse is rarely a single step. Instead, researchers use mathematical schedules to modulate the "pressure" of sparsity.
- Cubic and Exponential Schedules: In the seminal work To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression, Michael Zhu and Suyog Gupta introduced a cubic sparsity schedule. This prunes the network rapidly when redundancy is high and slows down as the model becomes sparser, giving the remaining weights time to adapt. Similarly, Barry Dong uses an asymptotic schedule in Progressive Deep Neural Networks Acceleration via Soft Filter Pruning to allow active parameters to progressively compensate for the loss of information.
- Smooth Annealing: For activation sparsity in Large Language Models, Chenyang Song and Xu Han propose ProSparse, which uses a sine-curve schedule to increase regularization pressure. This prevents the sudden collapse of internal feature distributions that often occurs with aggressive one-step "ReLUfication."
Soft Masking and Scaffolding
A key mechanism for making this transition differentiable is the use of "soft" masks. Instead of abruptly zeroing out a weight or feature, the model applies a continuous mask that gradually decays toward zero.
In Experimental Design for Multi-Channel Imaging via Task-Driven Feature Selection, Hongxiang Lin and Yukun Zhou use soft masking to slowly remove imaging channels. This prevents the gradient spikes associated with sudden feature removal, allowing the task-specific network to adapt its weights as the input dimensionality decreases.
Other approaches use temporary "scaffolding." Tianlong Chen in Training Your Sparse Neural Network Better with Any Mask introduces "ghost" neurons and auxiliary skip connections that exist only during the early phases of training to maintain gradient flow, before being phased out to leave a strictly sparse final architecture.
Stability in Mixture-of-Experts
The challenge of sparsity is particularly acute in Mixture-of-Experts (MoE) models, where "routing collapse" can occur if the model prematurely decides to send all tokens to a single expert.
To combat this, Xiao Liang and colleagues utilized a "progressive sparsification" schedule for Sigma-MoE-Tiny, starting with many active experts and reducing them to a target ratio. Lauren. A Hannah describes this in MoE-PHDS as "curriculum anchoring," which ensures that experts learn to function effectively even when fewer of them are co-activated for a given token.
Anchoring to the Past
Finally, to prevent "catastrophic forgetting" during the transition to extreme sparsity, some methods use the dense model as a guide. Runxin Xu and Fuli Luo developed ContrAstive Pruning (CAP), which uses intermediate snapshots of the model as training anchors. By forcing the sparse model to remain "close" to its slightly-less-sparse previous versions, the training process follows a smooth optimization path that preserves performance even at 97% sparsity.
By treating sparsity not as a static constraint but as a destination reached through a carefully managed curriculum, these techniques allow models to achieve the efficiency of sparse architectures with the stability and performance of dense ones.