Streamlining Tensor and Network Pruning in PyTorch

Michela Paganini, Jessica Zosa FordearXiv 2020 · arXiv

Abstract

This paper introduces torch.nn.utils.prune, a native PyTorch module designed to standardize and simplify neural network pruning. By providing a unified interface and extensible base classes, it lowers the technical barrier for researchers and engineers to compress models for efficient on-device deployment.

Introduction

Modern deep learning models are notoriously over-parametrized, leading to an "explosion in size" that complicates deployment on resource-constrained devices like smartphones, IoT sensors, or AR/VR hardware. While biological brains are remarkably sparse and efficient, artificial networks often rely on dense, redundant connections.

Pruning—the process of removing unnecessary parameters—is a primary strategy for model compression. However, prior to this work, implementing pruning in PyTorch was often a fragmented process. Researchers had to write custom, non-standardized code to handle masks, track original weights, and manage iterative pruning schedules. This lack of a common framework hindered reproducibility and slowed the transition from research to production.

Methodology

The torch.nn.utils.prune module centers on several key architectural choices designed to make pruning non-destructive and highly flexible:

  • Reparametrization: Instead of deleting weights outright, the module renames the original parameter (e.g., weight becomes weight_orig) and creates a binary weight_mask.
  • Forward Hooks: To maintain "live" pruning during training or inference, the module utilizes a forward_pre_hook. Every time the model is called, this hook computes the element-wise product of the original weight and the mask on the fly.
  • The BasePruningMethod Class: This abstract base class allows users to define custom pruning logic by simply overriding the compute_mask method.
The architectural transition from a standard parameter flow to a masked reparametrization after pruning is applied.

Advanced Pruning Capabilities

Beyond simple weight removal, the framework supports complex workflows:

  • Iterative Pruning: Through a PruningContainer, the module can handle multiple sequential pruning calls on the same tensor, allowing users to combine different techniques (e.g., initial random pruning followed by magnitude-based pruning).
  • Global Pruning: One of the most powerful features is the ability to perform global pruning. Instead of pruning a fixed percentage of each layer, users can prune the bottom $N$% of weights across the entire network, allowing the algorithm to automatically identify which layers contain the most redundancy.
  • Permanent Pruning: Once a pruning schedule is complete, users can call prune.remove(), which incorporates the mask into the weights permanently and removes the hooks to eliminate runtime overhead.

Discussion and Limitations

While the module standardizes the mathematical application of sparsity, it does not automatically translate to hardware speedups. Realizing latency benefits on standard CPUs/GPUs typically requires specialized kernels or hardware optimized for sparse computation, which are beyond the scope of this utility module.

Future work may involve extending the module to support more diverse pruning criteria, such as gradient-based or activation-based importance metrics, and improving the PruningContainer to handle more complex mask combination logic automatically. By moving pruning into the core utility library, PyTorch makes model compression as accessible as any other hyperparameter optimization.

Summary updated

Abstract

In order to contrast the explosion in size of state-of-the-art machine learning models that can be attributed to the empirical advantages of over-parametrization, and due to the necessity of deploying fast, sustainable, and private on-device models on resource-constrained devices, the community has focused on techniques such as pruning, quantization, and distillation as central strategies for model compression. Towards the goal of facilitating the adoption of a common interface for neural network pruning in PyTorch, this contribution describes the recent addition of the PyTorch torch.nn.utils.prune module, which provides shared, open source pruning functionalities to lower the technical implementation barrier to reducing model size and capacity before, during, and/or after training. We present the module's user interface, elucidate implementation details, illustrate example usage, and suggest ways to extend the contributed functionalities to new pruning methods.
BibTeX
@misc{paganini2020streamlining,
  author = {Michela Paganini and Jessica Zosa Forde},
  title = {{Streamlining Tensor and Network Pruning in PyTorch}},
  year = {2020},
  eprint = {2004.13770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.13770},
}
Figure