Structured Bayesian Pruning via Log-Normal Multiplicative Noise

Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, Dmitry Vetrov

Abstract: Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural networks and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is computed in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.

Introduction: Pruning with a Purpose

In deep learning, "sparse" is usually a synonym for "efficient." However, there is a catch: most pruning methods result in unstructured sparsity—where individual weights are zeroed out randomly. While this looks great on paper for compression, it does almost nothing to speed up computation on modern hardware like GPUs, which are optimized for dense matrix operations. To achieve real-world acceleration, one needs structured sparsity, which involves removing entire neurons or convolutional channels.

In their NIPS 2017 paper, Structured Bayesian Pruning via Log-Normal Multiplicative Noise, Neklyudov et al. introduce a method to achieve this using a clever Bayesian framework. By injecting log-normal noise into neuron outputs, they automatically identify and remove redundant components during training.

Methodology: The SBP Layer

The core of the proposed Structured Bayesian Pruning (SBP) is moving the noise from the weights to the neuron outputs.

  1. The SBP Layer: The authors introduce a dropout-like layer where each neuron's output is multiplied by a noise variable $\theta$. If the model learns that a neuron's noise should have a very low Signal-to-Noise Ratio (SNR), that neuron is effectively useless and can be pruned.
  2. Log-Normal Multiplicative Noise: Unlike standard Gaussian dropout, they use a log-normal distribution. This choice is crucial because it is strictly non-negative, ensuring it won't flip the sign of activations in ReLU-based networks, and it maps naturally to the log-uniform prior used to induce sparsity.
  3. Truncated Distributions: To avoid mathematical pitfalls like improper priors that lead to infinite values during optimization, the noise is restricted to a specific range (e.g., $[10^{-8}, 1]$) using truncated distributions. This ensures numerical stability.
  4. Variational Inference: The model is optimized using the Evidence Lower Bound (ELBO). This allows the network to "learn" which neurons to keep by balancing data fit against a penalty for keeping neurons active.
Figure 2: Training dynamics showing the evolution of the ELBO and the percentage of retained units over epochs.

Key Insights

  • SNR as a Pruning Metric: Instead of a simple dropout rate, the authors use the Signal-to-Noise Ratio (SNR). If a neuron's noise is so loud that the SNR drops below 1, the neuron is pruned. This provides a clear, theoretically grounded threshold for model compression.
  • The "No Prior Gap": By using a log-normal approximation for a log-uniform prior, the model can perfectly represent the "no data" state. This means the model is highly efficient at defaulting to sparsity unless the data strongly suggests a neuron is necessary.
  • Robustness to Random Labels: In a fascinating experiment, the authors showed that SBP doesn't overfit even when trained on completely random labels. While standard networks would memorize the noise, SBP simply prunes almost everything, realizing there is no useful signal to learn.

Experimental Results

The authors evaluated SBP on standard computer vision benchmarks including MNIST (LeNet architectures) and CIFAR-10 (VGG-like architectures). The results demonstrate that significant speedups are possible with negligible loss in accuracy.

Figure 3: Performance comparison on MNIST architectures, showing significant FLOP reductions and CPU speedups.
  • LeNet-500-300 (MNIST): Achieved an 11.23x reduction in FLOPs and a 2.33x speedup on CPU, reducing the hidden layers from 500-300 down to roughly 245-160 neurons.
  • LeNet-5-Caffe (MNIST): Achieved a 10.49x reduction in FLOPs and a 5.41x speedup on CPU.
  • VGG-like (CIFAR-10): The model pruned massive amounts of the network. In the later layers of VGG, it reduced 512 channels down to just 9 or 21, while maintaining accuracy within ~0.3% of the original dense model.

Limitations and Future Work

While the FLOP reduction is massive, the actual wall-clock speedup on GPUs is often lower than on CPUs. This is because GPUs require specific alignment and block-sizes to maximize throughput; simply removing a few filters might not always hit those hardware "sweet spots." Additionally, while the truncation parameters $a$ and $b$ are robust, the choice of the SNR threshold (set to 1) is a heuristic that might need adjustment for different tasks. Future work could extend this to more complex structures, like pruning entire blocks in Residual Networks (ResNets).

Summary updated

Abstract

Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural networks and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is computed in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.
BibTeX
@inproceedings{neklyudov2017structured,
  author = {Kirill Neklyudov and Dmitry Molchanov and Arsenii Ashukha and Dmitry Vetrov},
  title = {{Structured Bayesian Pruning via Log-Normal Multiplicative Noise}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=HJZ0uP-dZS},
}
Figure