Are Straight-Through Gradients and Soft-Thresholding All You Need for Sparse Training?

Antoine Vanderschueren, Christophe De VleeschouwerWACV 2023

TL;DR

This paper introduces ST-3, a simple yet highly effective method for training sparse neural networks in a single pass. By combining straight-through gradient estimation with soft-thresholding, ST-3 allows weights to smoothly transition to zero, achieving state-of-the-art accuracy-sparsity trade-offs without the computational burden of iterative pruning methods.

What Problem Does This Paper Address?

Training neural networks with billions of parameters is computationally expensive, both for training and inference. To reduce this cost, researchers aim to create "sparse" networks where many weights are set to zero. Traditional methods like pruning remove weights permanently, which can disrupt training and lead to accuracy loss, especially when weights need to change signs. More recent methods try to allow weights to become zero and then re-activate, but often require complex iterative training or sophisticated techniques. This paper addresses the challenge of achieving high sparsity with good accuracy in a single training cycle, avoiding the computational overhead of multi-cycle methods.

Methods

The core of ST-3 lies in two key ideas:

  • Soft-Thresholding: Instead of abruptly setting weights to zero (hard-thresholding) when they fall below a certain magnitude, soft-thresholding gently pushes them towards zero. This smoother transition prevents sharp discontinuities in the network's forward pass, which can destabilize training and hurt accuracy, especially at high sparsity levels.
  • Straight-Through Gradient Estimation (STE): To allow zeroed weights to become active again, ST-3 maintains a "raw" (non-thresholded) version of each weight. During the backward pass, the gradient is passed through as if the weight were still active, even if it was zeroed in the forward pass. This is inspired by techniques used in training quantized neural networks.

Additionally, the paper incorporates:

  • Progressive Sparsity Increase: The sparsity ratio is gradually increased throughout training, rather than being fixed from the start. This allows the network to adapt more gracefully.
  • Weight Rescaling: To compensate for the magnitude loss introduced by soft-thresholding, weights are rescaled. This helps maintain the overall magnitude of neuron activations.

Datasets

The paper evaluates its method on the following datasets:

  • Cifar-10
  • Cifar-100
  • ImageNet

The experiments use standard network architectures like ResNet-20, VGG-11, WideResNet-34x2, ResNet-50, and MobileNetv1.

Key Insights and Ideas

  • Simplicity is Key: The paper strongly suggests that effective sparse training doesn't require overly complex formulations. The combination of STE and soft-thresholding, when applied progressively, is surprisingly powerful.
  • Smoothness Matters: The transition to sparsity is crucial. Abrupt changes (hard-thresholding) are detrimental, while smooth transitions (soft-thresholding) lead to more stable training and better accuracy.
  • Re-activatability is Essential: Allowing weights to move between active and inactive states is vital for achieving high sparsity without sacrificing accuracy. STE facilitates this by enabling gradients to flow even for zeroed weights.
  • Single-Cycle Advantage: The ability to achieve state-of-the-art results in a single training cycle significantly reduces computational cost compared to iterative methods.

Key Results

  • State-of-the-Art Accuracy/Sparsity: ST-3 achieves state-of-the-art accuracy-sparsity trade-offs on Cifar-10 and ImageNet, outperforming many more complex methods.
  • Competitive Accuracy/FLOPS: A variant of ST-3 (ST-3º) also shows competitive results in terms of accuracy versus FLOPS reduction, indicating good inference speedup.
  • Improved over Baselines: ST-3 consistently outperforms methods like GMP and STR, and is competitive with or superior to more recent methods like ProbMask and GraNet, especially in single-cycle training.
  • Hybrid Approach Success: Combining ST-3 with iterative methods like LRR (denoted LRR+ST-3) even defines a new state-of-the-art, highlighting the benefits of STE and soft-thresholding.

Limitations and Future Work

  • Computational Cost of LRR+ST-3: While LRR+ST-3 achieves the best accuracy, it inherits the high computational cost of LRR due to multiple training cycles.
  • Data Augmentation Impact: The paper notes that the benefits of LRR over ST-3 can be influenced by data augmentation, suggesting that longer training with more augmentation can sometimes compensate for simpler methods.
  • Layer-wise Sparsity: While ST-3 primarily considers global sparsity, a variant (ST-3º) was explored to bias sparsity towards layers that offer greater FLOPS reduction. Further research into optimal layer-wise sparsity allocation could be beneficial.
  • Future Work: The authors suggest that their findings imply that the key to effective sparsification lies in enabling weights to evolve smoothly across the zero state while progressively increasing sparsity. This simple yet effective approach provides a strong baseline for future research in sparse neural networks.

Summary updated

BibTeX
@inproceedings{vanderschueren2023are,
  author = {Antoine Vanderschueren and Christophe De Vleeschouwer},
  title = {{Are Straight-Through Gradients and Soft-Thresholding All You Need for Sparse Training?}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure