TL;DR
This paper introduces Structured RigL (SRigL), a novel Dynamic Sparse Training (DST) method that learns structured sparsity with a constant fan-in constraint. SRigL achieves state-of-the-art generalization performance comparable to unstructured DST methods while enabling significant real-world inference speedups on CPUs and GPUs due to its structured sparsity.
What Problem Does This Paper Address?
Neural network sparsity, where many weights are set to zero, promises reduced computational cost and memory footprint. Dynamic Sparse Training (DST) methods are particularly effective as they learn sparsity during training, often outperforming static pruning and maintaining generalization close to dense models. However, a major hurdle is translating theoretical sparsity benefits into real-world speedups. Unstructured sparsity, while achieving high compression ratios, is difficult to accelerate on hardware. Structured sparsity, on the other hand, offers better hardware acceleration but can sometimes lead to a drop in generalization performance. This paper aims to bridge this gap by developing a DST method that learns structured sparsity while preserving generalization.
Methods
The core contribution is Structured RigL (SRigL), a sparse-to-sparse DST method built upon the RigL framework. SRigL introduces two key innovations:
- Constant Fan-In Constraint: This constraint enforces that each neuron in a layer has the same number of incoming connections. This structural property is crucial for efficient hardware acceleration.
- Neuron Ablation: The authors observed that at very high sparsities, standard RigL tends to "ablate" (remove) entire neurons to maintain generalization. SRigL incorporates this neuron ablation mechanism, allowing it to dynamically reduce layer width while maintaining the constant fan-in constraint for the remaining neurons. This is controlled by a hyperparameter $\gamma_{sal}$, which defines the minimum percentage of salient weights required per neuron.
The method involves periodic mask updates where weights are pruned based on magnitude and regrown based on gradient magnitude, but with the added constraint of maintaining the constant fan-in and enabling neuron ablation.
Datasets
The authors evaluate their method on standard image classification benchmarks:
- CIFAR-10: Used for training ResNet-18 and Wide ResNet-22.
- ImageNet (ILSVRC-12): Used for training ResNet-50, MobileNet-V3, and Vision Transformer (ViT-B/16).
Key Insights and Ideas
- Structured Sparsity for Real-World Speedups: The paper strongly advocates for structured sparsity as the key to unlocking practical inference acceleration, especially on commodity hardware.
- Neuron Ablation as a DST Feature: The observation that high-sparsity DST methods implicitly ablate neurons to preserve generalization is a critical insight. SRigL formalizes this, showing it's beneficial for structured sparsity too.
- Constant Fan-In is Not Detrimental to Generalization: Theoretical analysis and empirical results suggest that the constant fan-in constraint, while structural, does not negatively impact generalization performance, and can even lead to slightly smaller output-norm variance.
- Condensed Representation for Efficiency: The structured sparsity learned by SRigL allows for a "condensed" representation of weights, which is more parameter and memory efficient and amenable to hardware acceleration.
Key Results
- State-of-the-Art Generalization: SRigL achieves generalization performance comparable to RigL (an unstructured DST method) across various architectures and sparsity levels, even up to 99% sparsity.
- Significant Real-World Acceleration:
- On CPU, SRigL's condensed representation achieves a 3.4x speedup over dense models and 2.5x speedup over unstructured (CSR) sparse layers for online inference at 90% sparsity.
- On GPU, SRigL achieves a 1.7x speedup over dense models and a remarkable 13.0x speedup over unstructured (CSR) sparse layers for batched inference (batch size 256) at 90% sparsity.
- Neuron Ablation is Crucial at High Sparsities: For SRigL, enabling neuron ablation significantly improves performance at high sparsity levels, matching RigL's ability to maintain generalization.
- Competitive with Other DST Methods: SRigL performs comparably to or better than other DST methods, including DSB, in terms of accuracy.
Limitations and Future Work
- Hardware Optimization: While the paper demonstrates real-world speedups with a naive PyTorch implementation and leveraging existing GPU kernels, the authors acknowledge that a fully optimized software or hardware implementation of their condensed representation would require significant engineering effort.
- Hyperparameter Sensitivity: While SRigL is generally robust, the ViT-B/16 architecture showed more sensitivity to the $\gamma_{sal}$ hyperparameter, suggesting further investigation into optimal hyperparameter tuning for different architectures.
- Exploring Other Structured Sparsity Schemes: The authors hope their work will motivate the exploration and implementation of additional fine-grained structured sparsity schemes.
- In-Time Overparameterization Rates: The paper mentions that a detailed investigation of modified hyperparameters for RigL proposed in Liu et al. (2021c) for SRigL is left to future work.
Summary updated