Introduction
Vision Transformers (ViTs) have established new state-of-the-art benchmarks across various computer vision tasks. However, their deployment on resource-constrained edge devices, such as smartphones and IoT sensors, is often prohibited by their massive parameter counts and high computational requirements. Model pruning is a standard technique used to mitigate these costs by removing redundant components.
Traditional pruning methods typically operate as "black boxes," relying on simple heuristics like weight magnitude or complex architectural searches that ignore the semantic importance of individual units. This lack of interpretability often leads to sub-optimal pruning where the relationship between the model's internal logic and the target object classes is obscured. X-Pruner addresses this by shifting the pruning criterion from weight size to class-specific contribution.
The X-Pruner Framework
X-Pruner introduces a novel framework that makes ViT pruning "explainable" by measuring how much individual model units contribute to predicting specific target classes.
The methodology centers on several key innovations:
- Explainability-Aware Masks: Instead of uniform pruning across a layer, X-Pruner assigns a differentiable, class-specific mask to every prunable unit (such as an attention head or a row in a linear layer). This allows the model to learn which units are critical for identifying specific classes, such as a "White Wolf" versus a "Sailboat."
- Optimization Constraints: To ensure robustness, the framework incorporates a smoothness constraint—ensuring similar images produce similar mask activations—and a sparsity constraint to encourage the selection of only the most essential units.
- Differentiable Pruning Operation: The framework uses a specialized mathematical function to "softly" decide which units to retain based on a learned threshold. Because this operation is differentiable, the model can learn the optimal threshold for every layer simultaneously during training.
- Automated Layer-wise Rates: Utilizing an augmented Lagrangian method, X-Pruner automatically determines the best pruning rate for each layer to meet a global target, such as a 50% reduction in total FLOPs.
Experimental Evaluation
The researchers evaluated X-Pruner on standard benchmarks, including ILSVRC-12 (ImageNet) and CIFAR-10, across multiple architectures like DeiT (Data-efficient Image Transformers) and Swin Transformers.
Key Results
X-Pruner consistently outperformed state-of-the-art (SOTA) methods:
- DeiT-Small: Reduced FLOPs by 51.3% while maintaining a Top-1 accuracy of 79.04% (a minor 0.76% drop from the baseline).
- DeiT-Tiny: Achieved a massive 66.1% FLOP reduction with only a 1.1% accuracy loss.
- Swin-Tiny: Realized a 28.9% FLOP saving with a negligible 0.5% accuracy drop.
Interpretability and Qualitative Analysis
A significant advantage of X-Pruner is its impact on model interpretability. While traditional pruning often "breaks" a model's internal logic, visualizations using Layer-wise Relevance Propagation (LRP) show that X-Pruner results in cleaner and more focused attention maps compared to other methods.
Analysis of the learned masks revealed that higher layers in Transformers (closer to the output) generally retain higher mask values. This suggests that while lower layers handle local details that can be simplified, higher layers capture the rich, global semantic features essential for final predictions. By leveraging class-level information, the pruning process becomes more surgical and preserves the model's explanatory power.
Conclusion and Future Work
X-Pruner demonstrates that explainability can be a powerful tool for model compression. By identifying class-relevant units, it achieves superior efficiency without sacrificing accuracy or interpretability. Future work may extend this logic to more complex tasks like object detection or image segmentation and focus on benchmarking actual latency gains on specific edge hardware.
Summary updated