ABCP: Automatic Block-wise and Channel-wise Network Pruning via Joint Search

Jiaqi Li, Haoran Li, Yaran Chen, Zixiang Ding, Nannan Li, Mingjun Ma, Zicheng Duan, Dongbing Zhao

TL;DR

ABCP (Automatic Block-wise and Channel-wise Network Pruning) is a novel framework that uses deep reinforcement learning to simultaneously optimize both network depth and width. By jointly searching a discrete space to remove redundant residual blocks and a continuous space to determine exact channel pruning ratios, it compresses models like YOLOv3 by up to 99.5% with minimal accuracy loss—and sometimes even improves performance.

What Problem Does This Paper Address?

Deploying state-of-the-art deep learning models on resource-constrained edge devices (like mobile robots or drones) is notoriously difficult due to high computational and memory costs. Network pruning helps by stripping away redundant weights, but traditional rule-based approaches require tedious manual tuning and often yield suboptimal compression.

While automated pruning methods using reinforcement learning have emerged, most only focus on channel-wise pruning (shrinking layer width) while ignoring block-wise pruning (skipping entire residual blocks to reduce network depth). Furthermore, many existing automated tools rely on rigid, discrete search spaces (e.g., pruning in fixed increments of 10%), missing the fine-grained optimization required for highly sensitive layers. ABCP addresses these gaps by unifying depth and width reduction into a single automated search pipeline.

Methods

ABCP frames network pruning as a reinforcement learning problem solved via the REINFORCE policy gradient algorithm.

The core innovation is the Joint Sample Algorithm, which utilizes a stacked Long Short-Term Memory (LSTM) network to generate a complete pruning blueprint ($a_{1:T}$) for the target model:

  • Discrete Sampling for Depth: One branch of the LSTM cell determines whether an entire residual block should be kept or completely pruned ($a_{bi} \in \{0, 1\}$).
  • Continuous Sampling for Width: If a block is kept, a second branch samples the exact channel pruning ratio ($a_{li} \in [0, 0.9]$) from a continuous Gaussian distribution.
Fig. 2. The process of the pruning action sampling. (a) The model of the joint sample algorithm; (b) The structure of the residual network that will be pruned. The LSTM network is used for the joint sample algorithm where each cell is connected by two branches: one branch outputs the block pruning choice and the other branch outputs the layer pruning ratio. We denote four types of the layers in this residual network, so there are also four types of LSTM cells: the LSTM cell for the ordinary convolutional layer, the LSTM cell for the 1st layer of a residual block, the LSTM cell for the 2nd layer of a residual block, and the LSTM cell for the 1st layer of a residual group. During the pruning action sampling, each LSTM cell samples the corresponding block pruning choice abi or the layer pruning ratio ali for each block and each layer in the residual network respectively, which constitutes the list a1:T as the pruning action for the whole residual network.

A reward function balances accuracy and efficiency: $R = -L_{test} - \mathcal{F}/\lambda$, where $L_{test}$ is the validation loss after a quick 1-epoch fine-tuning step, $\mathcal{F}$ represents the model's FLOPs, and $\lambda$ is a user-defined trade-off hyperparameter. Once the DRL agent finds the optimal architecture blueprint, the pruned network is re-trained from scratch.

Datasets

The authors evaluated ABCP across three distinct object detection datasets:

  1. UCSD Dataset: A small, localized vehicle detection dataset (car, truck, bus) captured from freeway surveillance videos under varying traffic densities.
  2. Mobile Robot Detection Dataset: A newly introduced dataset collected via robot-mounted cameras containing 19,883 frames across 5 classes (red/blue robots, red/blue armor, and dead robots) featuring diverse exposures and angles.
  3. Sim2real Detection Dataset: A benchmark specifically designed to evaluate simulation-to-real-world transfer. It pairs a simulated environment dataset (generated via Gazebo) with real-world surveillance footage.

Key Insights and Ideas

  • The Synergy of Depth and Width: Coarse-grained block removal and fine-grained channel reduction are highly complementary. Optimizing both simultaneously prevents the search from getting trapped in local minima, which frequently happens in single-attribute or iterative rule-based pruning.
  • Continuous Search Spaces Matter: Modeling channel pruning ratios as a continuous distribution allows the agent to fine-tune layer widths precisely. The authors' ablation studies proved that restricting the search to discrete steps severely harms accuracy and limits compression.
  • Pruning as an Overfitting Antidote: For specialized or simpler tasks, massive models like YOLOv3 and YOLOv4 suffer from severe overfitting. Heavily pruning these networks actually boosted baseline accuracy because it stripped away the capacity to memorize background noise or simulation-specific artifacts.

Key Results

The quantitative results demonstrate massive efficiency gains with negligible—and sometimes positive—impacts on accuracy:

  • Mobile Robot Detection: On this dataset, ABCP reduced YOLOv3 FLOPs by 99.5% and parameters by 99.5%, resulting in a 37.3× inference speedup on an embedded GPU with only a 2.8% drop in mAP.
  • UCSD Dataset: The pruned model achieved 69.6% mAP, actually outperforming the baseline YOLOv3 (61.4% mAP) and YOLOv4 (63.1% mAP) while slashing FLOPs by 93.2%.
  • Sim2real Robustness: When trained on simulation data and tested directly on real-world data, the ABCP-pruned model achieved 76.1% mAP. This significantly outperformed baseline YOLOv3 (66.5% mAP) and crushed YOLOv4 (28.8% mAP), which failed due to severe overfitting to the simulator.

Limitations and Future Work

The primary limitation acknowledged in the paper is scope: ABCP was exclusively implemented and evaluated using the YOLOv3 architecture on object detection tasks. While the underlying joint-search logic is universally applicable to residual networks, the authors note that future work is required to validate and extend ABCP to other domains, such as image classification and 3D object detection.

Summary updated

Abstract

Currently, an increasing number of model pruning methods are proposed to resolve the contradictions between the computer powers required by the deep learning models and the resource-constrained devices. However, most of the traditional rule-based network pruning methods can not reach a sufficient compression ratio with low accuracy loss and are time-consuming as well as laborious. In this paper, we propose Automatic Block-wise and Channel-wise Network Pruning (ABCP) to jointly search the block-wise and channel-wise pruning action with deep reinforcement learning. A joint sample algorithm is proposed to simultaneously generate the pruning choice of each residual block and the channel pruning ratio of each convolutional layer from the discrete and continuous search space respectively. The best pruning action taking both the accuracy and the complexity of the model into account is obtained finally. Compared with the traditional rule-based pruning method, this pipeline saves human labor and achieves a higher compression ratio with lower accuracy loss. Tested on the mobile robot detection dataset, the pruned YOLOv3 model saves 99.5% FLOPs, reduces 99.5% parameters, and achieves 37.3 times speed up with only 2.8% mAP loss. The results of the transfer task on the sim2real detection dataset also show that our pruned model has much better robustness performance.
BibTeX
@article{li2023abcp,
  author = {Jiaqi Li and Haoran Li and Yaran Chen and Zixiang Ding and Nannan Li and Mingjun Ma and Zicheng Duan and Dongbing Zhao},
  title = {{ABCP: Automatic Block-wise and Channel-wise Network Pruning via Joint Search}},
  journal = {IEEE Trans. Cogn. Dev. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=nDvUtRW0f6},
}
Figure