NeurRev: Train Better Sparse Neural Network Practically via Neuron Revitalization

Gen Li, Lu Yin, Jie Ji, Wei Niu, Minghai Qin, Bin Ren, Linke Guo, Shiwei Liu, Xiaolong Ma

Overview

Training massive Deep Neural Networks (DNNs) is computationally expensive and memory-intensive, making it difficult to perform on resource-constrained "edge" devices like smartphones. Dynamic Sparse Training (DST) offers a solution by maintaining a sparse model throughout the entire training process—periodically pruning unimportant connections and "growing" new ones to find an optimal subnetwork.

However, existing DST methods have a major flaw: to be effective, they require very frequent topology updates. On mobile devices, each update triggers a costly recompilation of the computation graph, creating massive system overhead. Furthermore, researchers have discovered the existence of dormant neurons—neurons that consistently output zeros (due to the ReLU activation function) and stop learning entirely. Standard pruning techniques, which usually target weights with the smallest magnitudes, fail to remove the large negative weights that keep these neurons dormant.

The NeurRev Framework

To address these inefficiencies, the authors propose NeurRev (Neuron Revitalization), a framework that reimagines the pruning phase of DST:

  1. Dormant Neuron Identification: Instead of just looking at weight magnitudes, NeurRev monitors weight changes over time. If a neuron's weights aren't updating, it's likely dormant because its gradient flow is blocked.
  2. Surgical Pruning: Unlike traditional methods that prune small weights, NeurRev specifically identifies and prunes large negative weights that force a neuron's output to zero after the ReLU activation.
  3. Revitalization: By removing these "harmful" weights, the neuron is "awakened," allowing it to produce non-zero outputs and participate in the learning process again.
Illustration of the NeurRev pruning strategy targeting large negative weights to revitalize dormant neurons.
  1. System-Hardware Co-design: The framework is integrated with a compiler-based execution engine that generates optimized code for mobile CPUs and GPUs, specifically leveraging the stability of ReLU-based architectures.

Key Insights

  • The "Magnitude Pruning" Paradox: Traditional wisdom dictates pruning the smallest weights. NeurRev argues that in sparse networks, large negative weights are actually more "harmful" because they can permanently silence a neuron, wasting its potential expressive power.
  • Pruning as an "Awakening" Tool: Most DST research focuses on how to grow new weights effectively. NeurRev shows that optimizing the pruning step to maintain neuron health is just as critical for training stability.
  • Stability at Low Update Frequencies: Because NeurRev produces "healthier" neurons that learn more effectively, the model doesn't need to change its topology nearly as often. This enables practical training on mobile devices by avoiding constant recompilation.

Experimental Results

The researchers evaluated NeurRev across standard benchmarks (CIFAR-10/100, ImageNet-1K) using architectures like ResNet and VGG. Real-world system overhead and speedup tests were conducted on a Samsung Galaxy S21.

  • Fewer Dormant Neurons: NeurRev significantly reduces the number of all-zero feature maps during training compared to baseline DST methods like RigL, proving its ability to keep neurons active.
Comparison of all-zero feature maps between NeurRev and RigL over training epochs.
  • Superior Accuracy: NeurRev consistently outperformed representative state-of-the-art DST methods. On ResNet-32 (CIFAR-100) at 90% sparsity, NeurRev achieved 70.87% accuracy, surpassing the next best method by a significant margin.
  • Massive Overhead Reduction: On mobile devices, NeurRev required 12–96× fewer updates than competitors. This translated to a maximum of 112× less system overhead compared to other sparse training methods.
  • End-to-End Speedup: The framework achieved 1.8–2.1× acceleration compared to standard dense training on mobile hardware, proving that sparse training can be practically faster, not just theoretically more efficient.

Limitations and Future Work

The current rationale for "revitalization" is heavily tied to the behavior of the ReLU activation function (which zeros out negative values). While the principle could extend to other functions, the specific logic is optimized for ReLU. Additionally, the "update ratio" requires careful tuning at extreme sparsity levels (above 95%). Future work could explore applying the NeurRev principle to Transformers or other emerging architectures where sparsity is becoming increasingly important.

Summary updated

Abstract

Dynamic Sparse Training (DST) employs a greedy search mechanism to identify an optimal sparse subnetwork by periodically pruning and growing network connections during training. To guarantee effectiveness, DST algorithms rely on high search frequency, which consequently, requires large learning rate and batch size to enforce stable neuron learning. Such settings demand extreme memory consumption, as well as generating significant system overheads that limit the wide deployment of deep learning-based applications on resource-constraint platforms. To reconcile such, we propose $\underline{Neur}$on $\underline{Rev}$italization framework for DST (NeurRev), based on an innovative finding that dormant neurons exist with the presence of weight sparsity, and cannot be revitalized (i.e., activated for learning) even with high sparse mask search frequency. These dormant neurons produce a large quantity of zeros during training, which contribute relatively little to the outputs of succeeding layers or to the final results. Different from most existing DST algorithms that spare no effort designing weight growing criteria, NeurRev focuses on optimizing the long-neglected pruning part, which awakes dormant neurons by pruning and incurs no additional computation costs. As such, NeurRev advances more effective neuron learning, which not only achieves outperforming accuracy in a variety of networks and datasets, but also promoting a low-cost dynamism at system-level. Systematical evaluations on training speed and system overhead are conducted on the mobile devices, where the proposed NeurRev framework consistently outperforms representative state-of-the-arts. Code will be released.
BibTeX
@inproceedings{li2024neurrev,
  author = {Gen Li and Lu Yin and Jie Ji and Wei Niu and Minghai Qin and Bin Ren and Linke Guo and Shiwei Liu and Xiaolong Ma},
  title = {{NeurRev: Train Better Sparse Neural Network Practically via Neuron Revitalization}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=60lNoatp7u},
}
Figure