Probabilistic Fine-Tuning of Pruning Masks and PAC-Bayes Self-Bounded Learning
Abstract
Pruning neural networks typically relies on heuristic "one-shot" rules like magnitude-based pruning, which often yield sub-optimal masks. This paper proposes Probabilistic Fine-Tuning (PFT) to learn optimal pruning masks by optimizing keep-probabilities via the Gumbel-Softmax trick. Additionally, the authors introduce PAC-Bayes Pruning (PBP) to derive sparse networks with tight, non-vacuous generalization guarantees. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 demonstrate that PFT consistently improves accuracy at high sparsity levels, while PBP provides built-in certificates of performance.
Introduction
Modern deep neural networks are often over-parameterized, containing millions of redundant parameters. Pruning—the process of zeroing out unimportant weights—is essential for deploying these models on resource-constrained edge devices. However, traditional methods use "one-shot" heuristics (e.g., Magnitude Pruning or SNIP) to decide which weights to cut. These heuristic masks are rarely optimal and often require extensive weight fine-tuning after application. This research addresses a fundamental question: What if we fine-tuned the mask itself? Furthermore, the authors explore whether these pruned models can offer formal, mathematically tight generalization guarantees.
Methodology
The researchers move away from deterministic "keep or kill" masks, instead treating pruning as a stochastic process.
- Stochastic Pruning Masks: Each weight $w_i$ is assigned a probability $\lambda_i$ of being retained. During training, the mask is sampled from these probabilities.
- Probabilistic Fine-Tuning (PFT): This two-stage algorithm first trains a dense model, then optimizes the keep-probabilities ($\lambda_i$) using the Gumbel-Softmax trick. This allows gradient descent to be performed through discrete mask choices. Finally, probabilities are thresholded to create a deterministic sparse model.
- PAC-Bayes Pruning (PBP): This method optimizes a "spike-and-slab" distribution to minimize a PAC-Bayes bound rather than just training loss. This results in a "self-bounded" learner—a model with a built-in certificate of its maximum expected error on unseen data.
- Theoretical Analysis: The authors prove that optimizing mask probabilities acts as a form of data-adaptive $L_1$ regularization, distinguishing it from the $L_2$ regularization effect of Dropout.
Experimental Results
The methods were evaluated on standard computer vision benchmarks (MNIST, Fashion-MNIST, and CIFAR-10) across various architectures, including LeNet-5 and ResNet-20.
- Accuracy Boost: PFT consistently improved the test accuracy of models pruned via SNIP or Magnitude Pruning. For ResNet-20 on CIFAR-10, PFT improved accuracy by roughly 1% on average across different sparsity levels.
- Performance at High Sparsity: At extreme sparsity levels (e.g., 99%), PFT maintained high performance where standard one-shot methods failed significantly.
Key Insights
- Heuristics are Sub-optimal: At high sparsity, the masks found by PFT share almost no weights with those found by standard heuristics. This suggests that traditional methods "miss" the best sparse structures entirely.
- Pruning and Robustness: Pruned networks are naturally more robust to random perturbations of their weights than dense networks. This robustness is exactly what PAC-Bayes theory measures, explaining why pruning and PAC-Bayes are a powerful combination for generalization.
- Non-Vacuous Bounds: By using a fraction of the training data to "warm up" the model before calculating the PAC-Bayes bound, the authors achieved theoretical error bounds that were often within a few percentage points of the actual observed test error.
Limitations and Conclusion
While PFT finds superior masks, the optimization process using Gumbel-Softmax is computationally intensive, requiring as much effort as standard training. Additionally, while PBP provides strong theoretical guarantees, its stochastic predictors still slightly underperform deterministic counterparts in raw accuracy. Future work involves scaling these probabilistic methods to massive Transformer models or ImageNet-scale datasets.
In conclusion, learning the pruning mask through PFT and leveraging PAC-Bayes theory provides a principled path toward creating efficient neural networks that are both high-performing and theoretically grounded.
Summary updated