Understanding the Impact of Dynamic Channel Pruning on Conditionally Parameterized Convolutions

Ravi Raju, Dibakar Gope, Urmish Thakker, Jesse G. Beu

Abstract

Recent trends have shown that deep learning models have become larger and more accurate at an increased computational cost, making them difficult to deploy for latency-constrained applications. Conditional execution methods address this increase in cost while preserving the accuracy of the model by only performing the most important computation on important features. In this paper, we analyze a recent method, Feature Boosting and Suppression (FBS), which dynamically assesses which channels contain the most important input-dependent features and prune the others based on a runtime threshold gating mechanism. FBS is able to dynamically prune convolution filters with little loss in accuracy at conservative pruning rates. However, at aggressive pruning rates FBS suffers from heavy accuracy loss in a similar way to aggressive static pruning, due to a low number of active filters and correspondingly narrower effective network per inference. Conditionally parameterized convolutions (CondConv) is another work in the conditional execution domain that increases performance at small computation cost through the generation of input-dependent expert filters. We discover that substituting standard convolutional filters with input-specific filters, as described in CondConv, we enable FBS to address this accuracy loss because CondConv expert filters are able to bolster the narrower network's reduced capacity and capture distinct features from various classes in a single composite filter as needed, making aggressive pruning appealing again. We test our FBS-pruned CondConv model on CIFAR-10 with a custom 9-layer CNN and demonstrate that we can achieve up to 47.2% savings in computational costs at iso-accuracy and 1.01% improvement in accuracy at iso-computational costs over the state-of-art FBS technique.
BibTeX
@inproceedings{raju2020understanding,
  author = {Ravi Raju and Dibakar Gope and Urmish Thakker and Jesse G. Beu},
  title = {{Understanding the Impact of Dynamic Channel Pruning on Conditionally Parameterized Convolutions}},
  booktitle = {Aichallengeiot@sensys},
  year = {2020},
  doi = {10.1145/3417313.3429381},
}

Loading related content...

Figure