TL;DR
Vision Transformers (ViTs) have traditionally been considered much harder to compress than convolutional networks, rarely sustaining pruning beyond 50% sparsity. Optimal ViT Surgeon (oViT) introduces an efficient, correlation-aware second-order pruning framework combined with specialized cyclic fine-tuning recipes, demonstrating for the first time that ViTs can achieve 75–80% unstructured sparsity with less than a 1% relative accuracy drop.
What Problem Does This Paper Address?
While Vision Transformers (ViTs) achieve state-of-the-art vision performance, their massive computational and memory footprints hinder deployment on edge devices and resource-constrained environments. Convolutional networks (CNNs) like ResNet-50 can easily be pruned to 80–90% sparsity with negligible accuracy loss, but prior unstructured pruning methods on ViTs (such as SViTE) degraded sharply past 50% sparsity.
The paper identifies two main bottlenecks causing this disparity:
- Severe one-shot degradation: ViTs suffer drastic accuracy collapse when weights are pruned without accounting for weight correlations.
- Difficult recovery: Standard CNN fine-tuning schedules and heavy data augmentations fail to help pruned ViTs recover, often destabilizing sparse training.
Methods
The authors introduce oViT, which tackles both the selection of pruned weights and post-pruning recovery:
- Correlation-Aware Second-Order Pruning: Built upon the classical Optimal Brain Surgeon (OBS) framework using empirical Fisher information. The authors prove theoretically (Theorem 1) that finding the optimal subset of weights to remove while tracking weight correlations is equivalent to a constrained sparse regression problem that preserves the relationship between dense weights and calibration sample gradients.
- Efficient Block-Diagonal Solver: Standard combinatorial weight selection requires combinatorial search or cubic matrix inversions. oViT approximates the Fisher matrix into independent blocks, solving weight removal and updates greedily in $O(d \cdot B^2)$ time and $O(d \cdot B)$ space (where $B$ is block size and $d$ is parameter count), yielding negligible runtime overhead over simpler approximations.
- Cyclic Learning Rate Schedules: Second-order approximations assume near-zero gradients (a well-optimized local minimum) when pruning occurs. oViT uses a cyclic linear schedule where the learning rate decays toward zero right before pruning, then resets high immediately after to enable rapid recovery.
- Capacity-Matched Augmentation: Smaller models (e.g., DeiT-Tiny) become destabilized under aggressive augmentations when sparsified; oViT adopts lighter regularization recipes for small models while retaining standard heavier recipes for larger architectures.
- Single-Run Sparsity Sweeps: Rather than launching separate training runs for each target sparsity, oViT gradually prunes and saves intermediate checkpoints across targets (e.g., 50%, 60%, 75%, 90%) in a single schedule.
Datasets
- ImageNet-1K: The primary benchmark used to evaluate image classification across DeiT (-Tiny, -Small, -Base), Swin, XCiT, EfficientFormer, ResNet-50D, and EfficientNetV2.
- COCO: Used for evaluating one-shot pruning on object detection via the transformer-based DETR detector.
- CommonsenseQA: Evaluated under the challenging SMC-Bench (Sparsity May Cry) benchmark using RoBERTa-Large (354M parameters) to test transferability to NLP reasoning tasks.
Key Insights and Ideas
- ViTs are not inherently incompressible: The consensus that ViTs lack compressibility was an artifact of suboptimal pruners and mismatched fine-tuning recipes, rather than an intrinsic architectural flaw.
- Weight correlations are essential: Diagonal or independent-weight approximations discard critical interaction terms. Modeling local correlations via gradient-matching prevents catastrophic accuracy collapse during one-shot pruning.
- Sparsity as a regularizer: At moderate levels (40–50%), pruning acted as regularizer on smaller models, leading to higher Top-1 accuracy than the original dense baselines (e.g., +1.5% on DeiT-Tiny).
Key Results
- Matching CNN Compressibility: On DeiT-Tiny, oViT attains 73.7% accuracy at 50% sparsity (dense baseline: 72.2%) and retains 72.3% at 75% sparsity. At 90% sparsity, oViT reaches 68.0% (post fine-tuning), whereas prior state-of-the-art SViTE collapsed to 49.7%.
- DeiT-Small & DeiT-Base: DeiT-Small achieves 79.0% Top-1 accuracy at 75% sparsity (dense: 79.8%) and 75.8% at 90% (vs. SViTE’s 70.1%). DeiT-Base maintains 81.1% accuracy at 80% sparsity (dense: 81.8%).
- Practical CPU Acceleration: Running sparse models on the DeepSparse CPU engine produced a 2.4× speedup on DeiT-Tiny with zero accuracy drop, and 1.5× speedups on DeiT-Small/Base.
- Multi-task & Cross-Domain Versatility:
- Object Detection: Achieved 0.38 bbox mAP at 50% one-shot sparsity on DETR (dense: 0.42; magnitude pruning collapsed to 0.16).
- Language Models (SMC-Bench): On RoBERTa-Large running CommonsenseQA, where standard pruning methods completely fail even at low sparsity, oViT retained 76.16% accuracy at 50% sparsity and 50.37% at 90% sparsity.
- Composite Compression: Seamlessly combines with 8-bit quantization (QAT), dynamic token pruning (DynamicViT), and hardware-friendly 2:4 semi-structured sparsity (79.0% on DeiT-Small).
Limitations and Future Work
- Compute Overhead: Although cheaper than training sparse networks from scratch (e.g., 440 epochs total vs. SViTE's 600 epochs), obtaining optimal high-sparsity models still requires hundreds of fine-tuning epochs.
- Structured Pruning Scope: oViT focuses primarily on unstructured weight sparsity (with brief exploration of 2:4 sparsity). Extending correlation-aware regression solvers directly to structured coarse-grained pruning (e.g., pruning full attention heads or MLP channels) remains an open direction.
- Metric Evaluation: The evaluation centers strictly on Top-1 validation accuracy and throughput, without analyzing the impact of high sparsity on algorithmic bias, calibration, or out-of-distribution robustness.
Summary updated