Deep learning relies on a striking paradox: to find a compact, efficient model for deployment, we typically begin by training a massive, overparameterized dense network, only to prune away 90% or more of its weights after the fact. While this dense-to-sparse pipeline yields efficient models for inference, the training phase itself remains an immense computational and memory bottleneck.
Sparse-to-sparse training bypasses this costly dense phase entirely. By initializing a network with a sparse topology and maintaining a strict parameter and computational budget from the very first epoch to final convergence, it shifts the efficiency gains of sparsity from inference-only to the entire model lifecycle.
The Core Paradox: How to Prune What You Haven't Learned
If we initialize a network with only 10% of its weights active, how can we ensure we choose the right 10% before training even begins?
Gadhikar et al. provide a foundational answer in Why Random Pruning Is All We Need to Start Sparse. They demonstrate theoretically and empirically that a randomly pruned sparse network contains a subnetwork capable of approximating the performance of a dense model, provided it is slightly wider by a logarithmic factor. This implies that we do not need a dense initialization to start learning.
However, a static random mask will eventually bottleneck performance because the initial random guess cannot perfectly predict the optimal routing of information. To overcome this, sparse-to-sparse training relies on Dynamic Sparse Training (DST). Instead of keeping the network topology fixed, the algorithm allows the sparse connectivity pattern to evolve dynamically throughout the optimization process.
The Mechanism: In-Time Over-Parameterization
The engine behind dynamic sparse training is a continuous drop-and-grow cycle. At regular intervals during training, the network evaluates its active connections, prunes the least important ones, and allocates that saved parameter budget to initialize new connections elsewhere in the network.
[ Sparse Initialization ] ──> [ Train Active Weights ]
▲ │
│ ▼
[ Grow New Connections ] <─── [ Prune Weak Weights ]
(Exploration via Gradients) (Exploitation via Magnitude)
This process balances exploration and exploitation:
- Exploitation: The network drops weights that have shrunk close to zero, ensuring that computational resources are not wasted on inactive components.
- Exploration: The network activates new connections in regions that show high potential, typically guided by gradient information.
Liu et al. formalize the intuition of why this works in Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training. They introduce the concept of In-Time Over-Parameterization (ITOP). While a dense model keeps all parameters active simultaneously, a dynamic sparse model explores a vast space of parameter combinations sequentially over time. At any single forward and backward pass, the model remains highly sparse, but across the entire training trajectory, it "covers" the parameter space effectively, allowing it to discover optimal subnetworks without ever incurring the peak memory footprint of a dense model.
To prevent the optimization from getting trapped in poor local minima, Huang et al. show in Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-off that explicitly tracking how rarely a weight has been used can guide the "growth" phase. By intentionally regenerating rarely explored connections alongside high-gradient ones, the network maintains the plasticity required to escape optimization traps.
True Sparsity vs. Simulated Sparsity
A major hurdle in sparse training research is the gap between theoretical FLOP reductions and actual hardware speedups. Many standard deep learning frameworks simulate sparsity by multiplying a dense weight matrix by a binary mask of zeros and ones. While this correctly tracks accuracy, it still forces the hardware to compute multiplications by zero, saving neither time nor memory.
To achieve true hardware-aligned efficiency, sparse-to-sparse training must utilize specialized data structures and structured constraints:
- Native Sparse Data Structures: Rather than masking dense matrices, algorithms can use formats like Compressed Sparse Row (CSR) to store and compute only non-zero values. As van der Wal et al. demonstrate in Multilayer perceptron ensembles in a truly sparse training context, native sparse training eliminates the computational overhead of zero-valued parameters entirely. This allows large-scale models to be trained on standard consumer CPUs without specialized dense matrix accelerators, a result echoed by Liu in Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware.
- Structured and Combinatorial Sparsity: Modern hardware accelerators (like GPUs) favor structured patterns over completely random, unstructured sparsity. To align with hardware constraints, methods like the one proposed by Chao et al. in Learning Best Combination for Efficient N: M Sparsity optimize learnable importance scores for fixed weight combinations (such as 2:4 or $N:M$ sparsity patterns). By avoiding full-matrix dense gradient calculations and optimizing combination-level choices, they achieve significant wall-clock speedups during the training phase itself.
Co-Adaptation and End-to-End Consistency
Why does training a model sparsely from scratch often yield better results than taking a pre-trained dense model and forcing it into a sparse format? The answer lies in co-adaptation.
When a network is trained under a strict sparsity constraint from day one, every layer learns to maximize its representational capacity given the sparse signal it receives from preceding layers. For example, Yuan et al. introduce this end-to-end paradigm to large language models in Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. They show that training a transformer with a sparse attention mechanism from the initial pre-training phase allows the rest of the network's components to adapt natively to the sparse attention patterns, preventing the severe performance degradation that occurs when full-attention models are compressed post-hoc.
Similarly, training-time sparsity must be mathematically consistent with inference-time operations to avoid a "training-test mismatch." Martins et al. demonstrate in Sparse Text Generation that using bounded, sparse transformations (like entmax) paired with Fenchel-Young losses forces the model to natively learn sparse probability distributions during training, matching the exact truncation heuristics used during text decoding.
Decentralized and Edge Scaling
The ultimate utility of sparse-to-sparse training shines in resource-constrained and distributed environments, where transferring or storing dense matrices is physically impossible.
In neuromorphic computing, Liu et al. show in Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype that enforcing extreme connectivity sparsity (down to 1.3%) allows deep networks to be trained entirely within the limited local memory of a single neuromorphic core, bypassing the traditional training bottleneck.
[ Central Server ]
▲
│ (Low-Bandwidth Sparse Masks Only)
▼
┌───────────┐ ┌───────────┐
│ Edge Node │ │ Edge Node │
│ (Sparse │ │ (Sparse │
│ Training) │ │ Training) │
└───────────┘ └───────────┘
This principle scales directly to distributed and federated learning. In standard federated learning, edge devices must continuously upload and download massive dense gradient matrices, consuming immense network bandwidth. Frameworks like DisPFL by Dai and FedDST by Vikalo et al. solve this by keeping the local client models strictly sparse throughout the entire learning process. Because the topology updates dynamically on-device and only sparse parameter masks are communicated, these methods drastically reduce both local compute requirements and peer-to-peer communication costs while maintaining global convergence.
Key Foundational Work
Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization (2019) — contributed 1 concept elements to this direction.
Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype (2018) — contributed 1 concept elements to this direction.