Introduction
Deep Neural Networks (DNNs) have achieved remarkable success in computer vision but remain computationally expensive for deployment on resource-constrained edge devices. Filter pruning has emerged as a primary technique to reduce model complexity. While Hard Filter Pruning (HFP) permanently removes filters, Soft Filter Pruning (SFP) offers more flexibility by zeroing out unimportant filters during training while allowing them to be updated and potentially "regrow."
However, SFP suffers from a critical issue: representation inconsistency. During training, the model operates as a full, reconstructed network (R-NN), but at inference, it functions as a pruned network (P-NN). This discrepancy creates a performance gap, as the model is optimized for a state it does not inhabit during deployment. This "identity crisis" leads to suboptimal feature extraction in the pruned state compared to the full-capacity training phase.
Methodology: The CR-SFP Framework
To address this gap, we propose Consistent Representation for Soft Filter Pruning (CR-SFP). This approach treats pruning as a mutual learning task between the R-NN and P-NN, ensuring that the pruned network's feature space remains aligned with the full network's capacity.
The framework consists of three core components:
- Dual-Branch Training: The R-NN and P-NN are treated as two branches sharing the same backbone parameters but utilizing separate classifier heads. This allows the model to learn features that are effective for both the full and pruned configurations simultaneously.
- Data Distortion: Each branch receives a differently augmented version of the same input image. This data-distortion operation encourages the model to learn more robust and invariant features, further bridging the representation gap.
- Bidirectional KL-Divergence Loss: A mutual learning objective forces the output probability distributions of both branches to be consistent. By minimizing the KL-divergence between the R-NN and P-NN, the pruned model learns to mimic the richer representations of the full model. A "stopgrad" operation is applied to stabilize training, alternating the target distribution between the two branches to prevent trivial solutions.
Post-training, the auxiliary classifier is discarded, leaving a compact P-NN for inference. This ensures zero additional computational cost at runtime compared to standard SFP.
Experimental Results
We evaluated CR-SFP on the ImageNet dataset using the ResNet family of architectures (ResNet18, ResNet34, and ResNet50). Our method consistently outperforms baseline SFP and other state-of-the-art pruning techniques.
Key findings include:
- ResNet18: CR-SFP achieved a 69.2% top-1 accuracy with a 41.8% reduction in FLOPs, representing a 2.1% improvement over standard SFP.
- ResNet50: The model reached 76.6% top-1 accuracy, surpassing several methods that require pre-trained weights (PT: Y), despite CR-SFP training from scratch (PT: N).
- Efficiency and Robustness: While CR-SFP requires two forward passes during training, the overhead is minimal (approx. 4ms per step). Furthermore, the performance gains are independent of the specific pruning criteria used (e.g., $\ell_2$-norm or Geometric Median), suggesting that maintaining representation consistency is a fundamental requirement for effective soft pruning.
Conclusion
CR-SFP bridges the gap between training and inference in soft filter pruning by ensuring consistent representations through mutual learning. By forcing the pruned network to maintain the representational power of the full network, we achieve significant accuracy improvements without increasing inference-time complexity. Future research may extend this consistency-based philosophy to other compression techniques such as network quantization and binarization.
Summary updated