Structured Bayesian Pruning via Log-Normal Multiplicative Noise

Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, Dmitry P. Vetrov

Overview

This paper introduces Structured Bayesian Pruning (SBP), a method that automatically thins out neural networks by removing entire neurons or convolutional channels during training. By injecting truncated log-normal noise into the network and using Bayesian inference to tune it, the authors achieve significant hardware acceleration (up to 11x fewer FLOPs) with negligible loss in accuracy.

The Problem: Unstructured vs. Structured Sparsity

Deep neural networks are notoriously over-parameterized. While "sparsity" is a common solution to reduce model size, most methods produce unstructured sparsity—randomly zeroing out individual weights. Unfortunately, modern hardware (CPUs and GPUs) is optimized for dense matrix operations; skipping a few random zeros doesn't actually speed up computation.

To get a real-world speedup, you need structured sparsity: removing entire rows, columns, or 3D filters. Previous methods for structured sparsity often required tedious manual tuning of hyperparameters or used "improper" mathematical priors that made the training process unstable.

Methodology: Structured Bayesian Pruning

The authors propose a new "SBP layer" that can be dropped into existing architectures. Here’s the breakdown of their approach:

  1. Multiplicative Noise: Instead of dropping neurons binary-style (on/off), they multiply the output of each neuron by a random noise variable $\theta$.
  2. The Log-Normal Innovation: They model this noise using a log-normal distribution. Unlike the standard Gaussian noise used in Variational Dropout, log-normal noise is always positive, which prevents it from flipping the sign of the signal—a crucial feature when working with common activation functions like ReLU.
  3. The Truncation Trick: To solve a common mathematical headache in Bayesian deep learning (infinite KL divergence with improper priors), the authors use truncated distributions. They restrict the noise to a specific range (e.g., $[0, 1]$). This ensures the math stays "closed-form" (solvable with a formula) and numerically stable for computers.
  4. Automatic Pruning: During training, the model automatically adjusts the noise level for each neuron. If a neuron becomes too noisy to be useful (low Signal-to-Noise Ratio), it is permanently pruned from the computation graph.

Key Insights

  • Structured vs. Unstructured: The core "aha" moment is shifting the Bayesian "sparsity" prior from individual weights to the outputs of groups (like channels). By sharing a single noise variable across a whole filter, the model is forced to decide if that entire filter is worth keeping.
  • Log-Space is Easier: Modeling noise as log-normal is equivalent to adding Gaussian noise in the log-space. This makes the variational inference much more intuitive and easier to optimize than trying to use Bernoulli (binary) distributions.
  • No "Prior Gap": Because the log-uniform distribution is a specific case of the log-normal distribution, the model can perfectly represent a "completely uncertain" state, allowing it to prune neurons without being biased by a poorly chosen starting point.
  • Robustness to Overfitting: Interestingly, the authors found that SBP prevents the network from memorizing random labels—a common failure mode for standard dropout. The Bayesian regularization is strong enough to force the model to choose the simplest possible explanation for the data.

Experimental Results

The researchers evaluated SBP on standard computer vision benchmarks, including MNIST (with LeNet architectures) and CIFAR-10 (with a VGG-like architecture). The results show that SBP is highly effective at "slimming down" networks for faster inference.

Performance on LeNet architectures
  • LeNet-500-300 (MNIST): Achieved an 11.23x reduction in FLOPs and a 2.33x CPU speedup while maintaining an error rate (1.55%) nearly identical to the original model (1.54%).
  • LeNet-5-Caffe (MNIST): Achieved a 10.49x FLOP reduction and a 5.41x CPU speedup.
Performance on VGG-like architectures
  • VGG-like (CIFAR-10): Successfully pruned hundreds of filters across 13 convolutional layers. For example, in a widened VGG model, they reduced the FLOP count by over 3x with only a small impact on accuracy.
  • Comparison: SBP consistently provided higher speedups than competing methods like Structured Sparsity Learning (SSL) or weight-wise Sparse Variational Dropout (SparseVD).

Limitations and Future Directions

  • Implementation Complexity: While the authors provide a "dropout-like layer," implementing Bayesian inference (specifically the Reparameterization Trick for truncated distributions) is slightly more complex than standard training.
  • Scaling to ImageNet: The paper focuses on MNIST and CIFAR-10. While the results on VGG are promising, the performance on massive-scale datasets like ImageNet—where structured pruning is most needed—was not discussed.
  • Hardware Specifics: The reported speedups are on specific CPU (Intel Xeon) and GPU (Tesla K40) hardware. Real-world gains can vary significantly depending on the specific BLAS libraries and hardware architecture used.

Summary updated

Abstract

Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves gener- alization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural net- works and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is com- puted in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.
BibTeX
@inproceedings{neklyudov2017structured,
  author = {Kirill Neklyudov and Dmitry Molchanov and Arsenii Ashukha and Dmitry P. Vetrov},
  title = {{Structured Bayesian Pruning via Log-Normal Multiplicative Noise}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=HJZ0uP-dZS},
}
Figure