Overview
Training massive Deep Neural Networks (DNNs) is computationally expensive and memory-intensive, making it difficult to perform on resource-constrained "edge" devices like smartphones. Dynamic Sparse Training (DST) offers a solution by maintaining a sparse model throughout the entire training process—periodically pruning unimportant connections and "growing" new ones to find an optimal subnetwork.
However, existing DST methods have a major flaw: to be effective, they require very frequent topology updates. On mobile devices, each update triggers a costly recompilation of the computation graph, creating massive system overhead. Furthermore, researchers have discovered the existence of dormant neurons—neurons that consistently output zeros (due to the ReLU activation function) and stop learning entirely. Standard pruning techniques, which usually target weights with the smallest magnitudes, fail to remove the large negative weights that keep these neurons dormant.
The NeurRev Framework
To address these inefficiencies, the authors propose NeurRev (Neuron Revitalization), a framework that reimagines the pruning phase of DST:
- Dormant Neuron Identification: Instead of just looking at weight magnitudes, NeurRev monitors weight changes over time. If a neuron's weights aren't updating, it's likely dormant because its gradient flow is blocked.
- Surgical Pruning: Unlike traditional methods that prune small weights, NeurRev specifically identifies and prunes large negative weights that force a neuron's output to zero after the ReLU activation.
- Revitalization: By removing these "harmful" weights, the neuron is "awakened," allowing it to produce non-zero outputs and participate in the learning process again.
- System-Hardware Co-design: The framework is integrated with a compiler-based execution engine that generates optimized code for mobile CPUs and GPUs, specifically leveraging the stability of ReLU-based architectures.
Key Insights
- The "Magnitude Pruning" Paradox: Traditional wisdom dictates pruning the smallest weights. NeurRev argues that in sparse networks, large negative weights are actually more "harmful" because they can permanently silence a neuron, wasting its potential expressive power.
- Pruning as an "Awakening" Tool: Most DST research focuses on how to grow new weights effectively. NeurRev shows that optimizing the pruning step to maintain neuron health is just as critical for training stability.
- Stability at Low Update Frequencies: Because NeurRev produces "healthier" neurons that learn more effectively, the model doesn't need to change its topology nearly as often. This enables practical training on mobile devices by avoiding constant recompilation.
Experimental Results
The researchers evaluated NeurRev across standard benchmarks (CIFAR-10/100, ImageNet-1K) using architectures like ResNet and VGG. Real-world system overhead and speedup tests were conducted on a Samsung Galaxy S21.
- Fewer Dormant Neurons: NeurRev significantly reduces the number of all-zero feature maps during training compared to baseline DST methods like RigL, proving its ability to keep neurons active.
- Superior Accuracy: NeurRev consistently outperformed representative state-of-the-art DST methods. On ResNet-32 (CIFAR-100) at 90% sparsity, NeurRev achieved 70.87% accuracy, surpassing the next best method by a significant margin.
- Massive Overhead Reduction: On mobile devices, NeurRev required 12–96× fewer updates than competitors. This translated to a maximum of 112× less system overhead compared to other sparse training methods.
- End-to-End Speedup: The framework achieved 1.8–2.1× acceleration compared to standard dense training on mobile hardware, proving that sparse training can be practically faster, not just theoretically more efficient.
Limitations and Future Work
The current rationale for "revitalization" is heavily tied to the behavior of the ReLU activation function (which zeros out negative values). While the principle could extend to other functions, the specific logic is optimized for ReLU. Additionally, the "update ratio" requires careful tuning at extreme sparsity levels (above 95%). Future work could explore applying the NeurRev principle to Transformers or other emerging architectures where sparsity is becoming increasingly important.
Summary updated