AntiDote: Attention-based Dynamic Optimization for Neural Network Runtime Efficiency

Fuxun Yu, Chenchen Liu, Di Wang, Yanzhi Wang, Xiang Chen

Abstract

Convolutional Neural Networks (CNNs) are often too resource-intensive for edge deployment. While static pruning offers a solution by permanently removing weights, it fails to account for the dynamic, input-dependent importance of feature map components. We present AntiDote, a dynamic optimization framework that utilizes neural network attention mechanisms to identify and prune redundant channels and spatial columns for every individual input. Combined with a novel "Targeted Dropout" training scheme, AntiDote achieves up to 54.5% FLOPs reduction with negligible accuracy loss, demonstrating that dynamic, multi-dimensional pruning is key to efficient deep learning.

Introduction

The deployment of state-of-the-art CNNs on resource-constrained devices necessitates aggressive model compression. Traditional static pruning methods remove insignificant filters globally, which is inherently sub-optimal as the importance of specific features often varies by input. Furthermore, existing dynamic methods predominantly focus on channel-wise pruning, overlooking the substantial redundancy present in the spatial dimensions of high-resolution feature maps.

The AntiDote Framework

AntiDote introduces a two-pronged approach to enable runtime efficiency through dynamic feature map pruning and robust training.

Dynamic Feature Map Pruning

During the inference phase, AntiDote calculates attention scores to generate binary masks for two types of redundancy:

  1. Channel Attention: The model computes average activations for each channel and retains only the top-$k$ most active channels for a given input.
  2. Spatial Column Pruning: A spatial "heat map" is generated by averaging across the depth of the feature map. Vertical columns with low importance are identified and masked out.

Unlike soft attention mechanisms that scale values, AntiDote employs hard binarization. This allows the system to completely skip computations for zeroed-out regions, translating directly into FLOPs savings.

Overview of the AntiDote dynamic pruning pipeline.

Training with Targeted Dropout (TTD)

To ensure the model remains accurate under aggressive pruning, AntiDote utilizes Training with Targeted Dropout (TTD). Unlike standard random dropout, TTD specifically drops components with the lowest attention scores during training. A Dropout Ratio Ascent strategy is employed, gradually increasing pruning aggressiveness to help the network learn to rely on its most salient features.

Empirical Results and Insights

The framework was evaluated on VGG16 and ResNet56 architectures using CIFAR-10, CIFAR-100, and ImageNet-100 datasets.

Efficiency and Accuracy

AntiDote consistently outperformed state-of-the-art static pruning methods. On ImageNet-100 with VGG16, the framework achieved a 54.5% reduction in FLOPs while actually improving accuracy by 0.9%, likely due to the regularizing effect of TTD. Across all tested configurations, FLOPs reductions ranged from 37.4% to 54.5%.

The Significance of Spatial Redundancy

A key finding of this work is the dominance of spatial redundancy in high-resolution images. On ImageNet, spatial column pruning accounted for 52.1% of the total FLOPs reduction, compared to only 2.4% from channel pruning. This suggests that for complex, high-resolution data, identifying where to look is more impactful than deciding which filters to activate.

Composition of channel vs. spatial redundancy across different datasets and models.

Conclusion and Future Work

AntiDote demonstrates that attention mechanisms are highly effective "significance evaluators" for dynamic pruning. By addressing both channel and spatial redundancy, it provides a superior trade-off between speed and precision compared to static baselines. Future research will focus on optimizing hardware kernels for the irregular memory access patterns typical of dynamic pruning and extending the framework to Transformer-based architectures.

Summary updated

BibTeX
@inproceedings{yu2020antidote,
  author = {Fuxun Yu and Chenchen Liu and Di Wang and Yanzhi Wang and Xiang Chen},
  title = {{AntiDote: Attention-based Dynamic Optimization for Neural Network Runtime Efficiency}},
  booktitle = {DATE},
  year = {2020},
  doi = {10.23919/date48585.2020.9116416},
}
Figure