X-Pruner: eXplainable Pruning for Vision Transformers

Lu Yu, Wei XiangCVPR 2023 · DOI · OpenReview

Introduction

Vision Transformers (ViTs) have established new state-of-the-art benchmarks across various computer vision tasks. However, their deployment on resource-constrained edge devices, such as smartphones and IoT sensors, is often prohibited by their massive parameter counts and high computational requirements. Model pruning is a standard technique used to mitigate these costs by removing redundant components.

Traditional pruning methods typically operate as "black boxes," relying on simple heuristics like weight magnitude or complex architectural searches that ignore the semantic importance of individual units. This lack of interpretability often leads to sub-optimal pruning where the relationship between the model's internal logic and the target object classes is obscured. X-Pruner addresses this by shifting the pruning criterion from weight size to class-specific contribution.

The X-Pruner Framework

X-Pruner introduces a novel framework that makes ViT pruning "explainable" by measuring how much individual model units contribute to predicting specific target classes.

The X-Pruner framework employs differentiable, class-aware masks during training to identify redundant units, enabling structured pruning for efficient testing.

The methodology centers on several key innovations:

  1. Explainability-Aware Masks: Instead of uniform pruning across a layer, X-Pruner assigns a differentiable, class-specific mask to every prunable unit (such as an attention head or a row in a linear layer). This allows the model to learn which units are critical for identifying specific classes, such as a "White Wolf" versus a "Sailboat."
  2. Optimization Constraints: To ensure robustness, the framework incorporates a smoothness constraint—ensuring similar images produce similar mask activations—and a sparsity constraint to encourage the selection of only the most essential units.
  3. Differentiable Pruning Operation: The framework uses a specialized mathematical function to "softly" decide which units to retain based on a learned threshold. Because this operation is differentiable, the model can learn the optimal threshold for every layer simultaneously during training.
  4. Automated Layer-wise Rates: Utilizing an augmented Lagrangian method, X-Pruner automatically determines the best pruning rate for each layer to meet a global target, such as a 50% reduction in total FLOPs.

Experimental Evaluation

The researchers evaluated X-Pruner on standard benchmarks, including ILSVRC-12 (ImageNet) and CIFAR-10, across multiple architectures like DeiT (Data-efficient Image Transformers) and Swin Transformers.

Key Results

X-Pruner consistently outperformed state-of-the-art (SOTA) methods:

  • DeiT-Small: Reduced FLOPs by 51.3% while maintaining a Top-1 accuracy of 79.04% (a minor 0.76% drop from the baseline).
  • DeiT-Tiny: Achieved a massive 66.1% FLOP reduction with only a 1.1% accuracy loss.
  • Swin-Tiny: Realized a 28.9% FLOP saving with a negligible 0.5% accuracy drop.

Interpretability and Qualitative Analysis

A significant advantage of X-Pruner is its impact on model interpretability. While traditional pruning often "breaks" a model's internal logic, visualizations using Layer-wise Relevance Propagation (LRP) show that X-Pruner results in cleaner and more focused attention maps compared to other methods.

Visual comparison of attention maps across different pruning methods. X-Pruner (bottom row) maintains the most coherent and focused attention on target objects.

Analysis of the learned masks revealed that higher layers in Transformers (closer to the output) generally retain higher mask values. This suggests that while lower layers handle local details that can be simplified, higher layers capture the rich, global semantic features essential for final predictions. By leveraging class-level information, the pruning process becomes more surgical and preserves the model's explanatory power.

Conclusion and Future Work

X-Pruner demonstrates that explainability can be a powerful tool for model compression. By identifying class-relevant units, it achieves superior efficiency without sacrificing accuracy or interpretability. Future work may extend this logic to more complex tasks like object detection or image segmentation and focus on benchmarking actual latency gains on specific edge hardware.

Summary updated

Abstract

Recently vision transformer models have become prominent models for a range of tasks. These models, however, usually suffer from intensive computational costs and heavy memory requirements, making them impractical for deployment on edge platforms. Recent studies have proposed to prune transformers in an unexplainable manner, which overlook the relationship between internal units of the model and the target class, thereby leading to inferior performance. To alleviate this problem, we propose a novel explainable pruning framework dubbed X-Pruner, which is designed by considering the explainability of the pruning criterion. Specifically, to measure each prunable unit's contribution to predicting each target class, a novel explainability-aware mask is proposed and learned in an end-to-end manner. Then, to preserve the most informative units and learn the layer-wise pruning rate, we adaptively search the layer-wise threshold that differentiates between unpruned and pruned units based on their explainability-aware mask values. To verify and evaluate our method, we apply the X-Pruner on representative transformer models including the DeiT and Swin Transformer. Comprehensive simulation results demonstrate that the proposed X-Pruner outperforms the state-of-the-art black-box methods with significantly reduced computational costs and slight performance degradation.
BibTeX
@inproceedings{yu2023x,
  author = {Lu Yu and Wei Xiang},
  title = {{X-Pruner: eXplainable Pruning for Vision Transformers}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02333},
}
Figure