Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs

Paul Wimmer, Jens Mehnert, Alexandru ConduracheCVPR 2022 · DOI · arXiv

TL;DR

Interspace Pruning (IP) shifts the pruning process from raw spatial weights to a learned, adaptive filter basis. By pruning coefficients in this "interspace" rather than individual pixels, sparse networks maintain better gradient flow and expressivity, leading to significantly higher accuracy and better generalization than standard pruning methods.

What Problem Does This Paper Address?

Pruning is a popular technique to reduce the memory and computational footprint of Convolutional Neural Networks (CNNs) by setting many weights to zero. However, standard unstructured pruning (SP)—which zeros out individual spatial weights—has a major drawback: it imposes a rigid, fixed subspace on the filters.

When you prune heavily before or during training (Pruning at Initialization or Dynamic Sparse Training), you often end up with "dead" filters that have too few active pixels to learn effectively. This leads to vanishing gradients and poor performance at high sparsity levels. The authors identify this as a "strong bias" that prevents sparse models from reaching their full potential.

Methods

The authors introduce Interspace Pruning (IP). Instead of treating a filter as a grid of $K \times K$ independent spatial weights, they represent it as a linear combination of a Filter Basis (FB).

  1. The Representation: A filter $h$ is modeled as $h = \sum \lambda_n g^{(n)}$, where $\{g^{(n)}\}$ is the Filter Basis and $\lambda_n$ are the coefficients.
  2. Adaptive Basis: Unlike standard pruning (which implicitly uses a fixed identity basis), IP treats the basis filters themselves as trainable parameters.
  3. Pruning the Coefficients: Pruning is applied to the coefficients ($\lambda$), not the spatial weights.
  4. Joint Training: The basis and the unpruned coefficients are trained jointly.

Because the basis filters are learned and can be spatially dense, the final reconstructed filter can remain expressive and dense even if it is represented by only a few non-zero coefficients. The authors also explore different sharing schemes, where a single basis can be shared across a whole layer, a block, or even the entire network to act as a regularizer.

Datasets

The researchers evaluated their method on standard computer vision benchmarks:

  • CIFAR-10: Using VGG16 architectures.
  • ImageNet (ILSVRC2012): Using ResNet18 and ResNet50 architectures.

They tested IP across a wide variety of state-of-the-art pruning frameworks, including SNIP, GraSP, SynFlow, RigL, SET, and Lottery Ticket (LT) methods.

Key Insights and Ideas

  • Dense Filters from Sparse Representations: The "aha moment" is that IP allows a model to be mathematically sparse (few parameters to store) while remaining spatially dense (expressive filters). This prevents the "vanishing gradient" problem common in standard sparse training.
  • Regularization through Sharing: Sharing a global basis across layers (coarse sharing) forces the network to find common visual primitives. This acts as a powerful regularizer, often leading to better generalization on the test set than dense models or standard sparse models.
  • Theoretical Superiority: The authors provide a mathematical proof (Theorem 1) showing that an adaptive basis will always provide a better approximation of a target filter than a fixed standard basis for any given sparsity level.
  • Trainability: IP-pruned networks show much more stable gradient norms during the early stages of training compared to SP-pruned networks, which often see their gradients collapse at high sparsity.

Key Results

  • Superior Accuracy: IP outperformed standard pruning across every method and dataset tested.
  • Efficiency Gains: On CIFAR-10, an IP-boosted Lottery Ticket model reached the same accuracy as the best standard pruning result while using 3.7x fewer parameters.
  • ImageNet Performance: Using IP with Fine-Tuning (IP-FT) on ResNet50 matched the performance of a standard dense model while training only 20% of the parameters.
  • Real-world Speedup: In CPU benchmarks, IP achieved a 5.2x speedup over dense training while maintaining better accuracy than standard pruning at the same runtime.

Limitations and Future Work

  • Computational Overhead: IP introduces a small constant computational overhead during the forward and backward passes due to the basis transformation. While this is offset by the ability to achieve higher accuracy at higher sparsity, it is a factor to consider for extremely latency-sensitive applications.
  • Hardware/Software Support: Like all unstructured pruning methods, IP requires specialized sparse kernels (like those in scipy.sparse) to realize actual speedups on commodity hardware.
  • Future Directions: The authors suggest combining IP with low-rank tensor approximations to further reduce costs. They also see potential in extending the "interspace" concept to other architectures like MLPs and Self-Attention modules in Transformers.

Summary updated

BibTeX
@inproceedings{wimmer2022interspace,
  author = {Paul Wimmer and Jens Mehnert and Alexandru Condurache},
  title = {{Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01220},
}
Figure