Optg: Optimizing Gradient-Driven Criteria in Network Sparsity

Yuxin Zhang, Mingbao Lin, Mengzhao Chen, Fei Chao, Yonghong Tian, Rongrong JiSSRN 2024 · DOI

TL;DR

This paper tackles the "independence paradox" in network sparsity, where pruning weights individually ignores their collective impact. They introduce OptG, a method that integrates supermask training with gradient-driven sparsity using a novel optimizer to effectively handle weight dependencies, leading to significant performance improvements, especially at high sparsity levels.

What Problem Does This Paper Address?

Deep neural networks, particularly Convolutional Neural Networks (CNNs), have grown enormously in size, making them challenging to deploy on resource-constrained devices. Network sparsity, the process of removing redundant parameters to create smaller, more efficient models, is a key solution. Many existing sparsity methods rely on "gradient-driven sparsity," which uses weight gradients to estimate the impact of removing a weight on the network's loss. However, these methods typically assume weights are independent, meaning removing one weight doesn't affect others. This assumption is flawed in practice, as weights are highly interdependent. This "independence paradox" leads to an "error gap," where the estimated loss change from pruning doesn't accurately reflect the actual loss change, especially when many weights are pruned simultaneously. This limits the performance of existing methods, particularly at high sparsity levels.

Methods

The paper proposes OptG, a novel approach that aims to overcome the independence paradox by integrating supermask training with gradient-driven sparsity.

Here's a breakdown of their approach:

  • Gradient-Driven Sparsity: This is the foundation, using the first-order Taylor expansion of the loss function to approximate the impact of removing a weight. The core idea is to remove weights that cause the least increase in training loss.
  • Supermask Training: This technique, which updates only mask values without modifying weights, is shown to implicitly accumulate gradient-driven sparsity criteria for both preserved and removed weights. The authors prove this mechanism can partially solve the independence paradox.
  • OptG's Novelty: OptG combines these two by interleaving supermask training into the during-training sparsity process. It introduces a novel supermask optimizer that:
  • Continuously accumulates mask gradients during training iterations.
  • Updates the binary mask only at the beginning of each training epoch. This allows the remaining weights to be finely tuned on the training set, mitigating the error gap.
  • Employs a sparsity-aware learning rate schedule for the masks. This schedule allocates learning rates proportionally to the current sparsity level, further reducing the impact of the independence paradox.
  • Utilizes a gradual sparsity schedule (Eq. 16) that increases sparsity smoothly over epochs, preventing sudden large error gaps.

Datasets

The paper evaluates OptG on the following datasets:

  • CIFAR-10/100: Used for training and evaluating VGGNet-19.
  • ImageNet: Used for training and evaluating ResNet-50 and MobileNet-V1.

No new benchmarks were introduced.

Key Insights and Ideas

  • The Independence Paradox is Real and Damaging: The paper provides a clear quantitative and qualitative explanation of why assuming weight independence in sparsity methods leads to suboptimal performance, especially when many weights are pruned at once.
  • Supermask Training is More Than Just Mask Updates: The authors demonstrate that supermask training, at its core, is implicitly performing gradient-driven sparsity, but in a way that better accounts for weight dependencies.
  • Fixed Weights in Supermask Training are a Bottleneck: While supermask training helps, keeping weights fixed prevents them from fully adapting to the pruned structure, leaving room for improvement.
  • OptG's Optimizer is Key: The novel supermask optimizer, by decoupling mask updates from weight updates within an epoch and using a sparsity-aware learning rate, effectively addresses the independence paradox.
  • Gradual Sparsity is Crucial: A smooth increase in sparsity over training epochs, rather than aggressive pruning, is shown to be beneficial for OptG's performance.

Key Results

OptG consistently outperforms state-of-the-art methods across various datasets and sparsity levels, particularly at ultra-high sparsity.

  • ResNet-50 on ImageNet: At 98% sparsity, OptG achieves 67.20% top-1 accuracy, significantly surpassing STR (62.84%). At 99% sparsity, OptG reaches 62.10%, outperforming GraNet by 4.02%.
  • VGGNet-19 on CIFAR-10/100: OptG shows negligible accuracy loss (0.01%) at 95% sparsity on CIFAR-10, compared to SNIP's 4.26% drop.
  • MobileNet-V1: OptG achieves 70.27% top-1 accuracy at 80% sparsity, outperforming STR by 3.75% with a more efficient parameter count.
  • Ablation Studies: The proposed gradual sparsity schedule and sparsity-aware mask learning rate schedule are shown to be critical for OptG's success.

Limitations and Future Work

  • Increased Training FLOPs: OptG requires more training FLOPs due to its specialized sparsity schedule. However, the authors note that speedups from irregular sparse tensors on common hardware are often negligible.
  • Performance at Low Sparsity: While OptG excels at high sparsity, its performance at relatively low sparsity rates could be further improved.
  • Limited to CNNs: The current work focuses on CNNs. The authors suggest future work should explore OptG's efficacy on Vision Transformer (ViT) models.

Summary updated

BibTeX
@misc{zhang2024optg,
  author = {Yuxin Zhang and Mingbao Lin and Mengzhao Chen and Fei Chao and Yonghong Tian and Rongrong Ji},
  title = {{Optg: Optimizing Gradient-Driven Criteria in Network Sparsity}},
  year = {2024},
  doi = {10.2139/ssrn.4829145},
  howpublished = {SSRN},
}
Figure