Abstract
We introduce Dirichlet pruning, a novel post-processing technique to transform a large neural network model into a compressed one. Dirichlet pruning is a form of structured pruning which assigns the Dirichlet distribution over each layer’s channels in convolutional layers (or neurons in fully-connected layers), and learns the parameters of the distribution over these units using variational inference. The learnt parameters allow us to informatively and intuitively remove unimportant units, resulting in a compact architecture containing only crucial features for a task at hand. This method yields low GPU footprint, as the number of parameters is linear in the number of channels (or neurons) and training requires as little as one epoch to converge. We perform extensive experiments, in particular on larger architectures such as VGG and WideResNet (94% and 72% compression rate, respectively) where our method achieves the state-of-the-art compression performance and provides interpretable features as a by-product.
Introduction
Modern neural networks have achieved unprecedented accuracy across various domains, but this success often comes at the cost of extreme over-parameterization. Large models are computationally expensive, memory-intensive, and difficult to deploy on edge devices. While pruning—the process of removing redundant parameters—is a common solution, traditional methods often rely on simple heuristics like weight magnitude (L1 or L2 norms) which may not accurately reflect a component's actual contribution to the task.
Furthermore, "unstructured" pruning, which removes individual weights, often fails to deliver real-world speedups on standard hardware due to the resulting sparse matrices. This paper proposes Dirichlet Pruning, a structured pruning framework that removes entire channels or neurons by modeling their relative importance through a principled probabilistic lens.
Methodology: The Dirichlet Importance Switch
The core of the proposed method is the introduction of an importance switch ($s_l$) for every layer $l$. Unlike standard architectures where the output is simply the activation of the hidden state, Dirichlet Pruning modifies the forward pass to include a probability vector that scales the hidden units.
The Dirichlet Prior and Variational Inference
The authors assign a Dirichlet distribution as a prior over these switches. As a distribution over distributions, samples from a Dirichlet distribution always sum to one. This mathematical property forces channels within a layer to "compete" for importance within a fixed budget, naturally producing a clear ranking of essential features.
Because calculating the true posterior of these switches is intractable, the method employs Variational Inference (VI) to estimate the parameters. During this phase, the original network weights are frozen, and only the Dirichlet parameters ($\phi$) are optimized. This approach is highly efficient, often requiring only a single epoch to converge.
Pruning Pipeline
- Importance Learning: Train the Dirichlet parameters to identify crucial channels.
- Architecture Selection: Rank channels based on the learned distribution and prune those with low importance.
- Fine-tuning: Perform a brief fine-tuning session to recover any minor accuracy loss in the now-slimmer model.
Experimental Results
The researchers evaluated Dirichlet Pruning on standard benchmarks including MNIST, FashionMNIST, and CIFAR-10, testing architectures ranging from LeNet-5 to VGG-16 and WideResNet.
- VGG-16 (CIFAR-10): The method achieved a massive reduction in parameters from approximately 15M to just 0.84M—a 94% compression rate—while significantly reducing FLOPs and maintaining competitive accuracy against state-of-the-art methods like Hrank.
- ResNet-56 (CIFAR-10): Parameters were reduced to 0.24M, outperforming recent structured pruning methods such as GAL and CP.
- LeNet-5 (MNIST): The model was compressed to just 5.5K parameters, significantly smaller than the 9K-15K range typically achieved by other Bayesian approaches.
Interpretability and Insights
A significant advantage of this probabilistic approach is the interpretability it provides. By visualizing the feature maps of the top-ranked channels, the authors observed that the pruned network retains channels that capture human-perceivable features, such as specific object silhouettes or structural curves, while discarding noisy or redundant information.
This "competition" induced by the Dirichlet distribution forces the model to be decisive, leading to a clearer separation between essential and non-essential features compared to standard norm-based pruning.
Conclusion and Future Work
Dirichlet Pruning offers a fast, efficient, and interpretable path toward model compression. By treating channel importance as a learned distribution rather than a fixed heuristic, it achieves state-of-the-art results with minimal computational overhead. Future research may explore using the posterior uncertainty of the Dirichlet distribution to automatically determine optimal pruning rates and extending the framework to modern architectures like Transformers.
Summary updated