ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks

Xiaohan Ding, Yuchen Guo, Guiguang Ding, Jungong Han

Abstract

Standard Convolutional Neural Networks (CNNs) typically utilize square convolution kernels, such as the ubiquitous $3 \times 3$ filter. However, research suggests that the central "skeleton" of these kernels—the horizontal and vertical cross—carries significantly more weight and importance for feature extraction than the corner pixels. ACNet introduces the Asymmetric Convolution Block (ACB), a novel architectural component designed to strengthen these skeletons during training. By employing a multi-branch structure during the learning phase and fusing these branches into a single standard kernel for deployment, ACNet provides a "free lunch": improved model accuracy with zero additional computational cost, memory footprint, or latency during inference.

Introduction

In deep learning, enhancing model performance often necessitates increasing the computational budget through deeper architectures or complex neural architecture searches. ACNet challenges this paradigm by focusing on the internal structure of the convolution kernel itself. The authors observed that the central crisscross pixels of a $3 \times 3$ kernel are more critical to performance than the corners. To exploit this, they proposed explicitly reinforcing this "skeleton" during training without burdening the final deployed model.

The Asymmetric Convolution Block (ACB)

The core of ACNet is the ACB, which replaces standard square convolution layers during the training phase.

Training Phase

Instead of a single $d \times d$ convolution, the ACB utilizes three parallel branches:

  1. A standard $3 \times 3$ kernel.
  2. A horizontal $1 \times 3$ kernel.
  3. A vertical $3 \times 1$ kernel.

The outputs of these three branches are summed and then passed through a Batch Normalization (BN) layer. This configuration allows the model to learn richer representations by specifically focusing on the horizontal and vertical features of the input.

The transition from a multi-branch training-time ACNet to a fused, single-kernel deployed model.

Inference and Deployment

Because convolution is a linear operation, the multi-branch structure used during training can be mathematically collapsed into a single standard kernel for inference. This process, known as fusion, ensures that the deployed model has the exact same architecture and speed as the original baseline, while retaining the performance gains from the asymmetric training.

Mathematical Fusion Mechanism

The fusion process occurs in two primary steps:

  1. BN Fusion: The parameters of the Batch Normalization layers (mean, variance, gamma, and beta) are "folded" into the weights and biases of their respective convolution kernels.
  2. Branch Fusion: The resulting $1 \times 3$ and $3 \times 1$ kernels are added to the corresponding central rows and columns of the $3 \times 3$ kernel.
The detailed fusion process: Batch Normalization parameters are folded into the weights before the branches are summed into a final kernel.

This mathematical equivalence allows the three-branch training block to be replaced by a single $3 \times 3$ convolution layer during deployment, maintaining the original model's efficiency.

Experimental Results

ACNet was evaluated across several benchmarks, including CIFAR-10, CIFAR-100, and ImageNet, using architectures like VGG, ResNet, and DenseNet. The results consistently showed performance improvements with zero inference overhead:

  • ImageNet Classification: AlexNet saw a Top-1 accuracy increase of 1.52%, while ResNet-18 and DenseNet-121 improved by 0.78% and 0.67%, respectively.
  • Rotational Robustness: ACNet-enhanced models demonstrated superior performance on images with rotational distortions. For example, on ImageNet images rotated by 180°, an ACNet-ResNet-18 outperformed the baseline by nearly 1%.
  • Hyper-parameter Free: A significant advantage of ACNet is that it requires no additional hyper-parameter tuning, making it a seamless drop-in replacement for standard convolution layers.

Conclusion and Limitations

ACNet provides a practical method for boosting CNN performance by strengthening kernel skeletons. While it introduces some additional computational overhead during the training phase, the "free" accuracy gains at inference time make it highly attractive for production environments. Future research may further explore the relationship between kernel spatial locations and rotational invariance to design even more efficient and robust architectures.

Summary updated

Abstract

As designing appropriate Convolutional Neural Network (CNN) architecture in the context of a given application usually involves heavy human works or numerous GPU hours, the research community is soliciting the architecture-neutral CNN structures, which can be easily plugged into multiple mature architectures to improve the performance on our real-world applications. We propose Asymmetric Convolution Block (ACB), an architecture-neutral structure as a CNN building block, which uses 1D asymmetric convolutions to strengthen the square convolution kernels. For an off-the-shelf architecture, we replace the standard square-kernel convolutional layers with ACBs to construct an Asymmetric Convolutional Network (ACNet), which can be trained to reach a higher level of accuracy. After training, we equivalently convert the ACNet into the same original architecture, thus requiring no extra computations anymore. We have observed that ACNet can improve the performance of various models on CIFAR and ImageNet by a clear margin. Through further experiments, we attribute the effectiveness of ACB to its capability of enhancing the model's robustness to rotational distortions and strengthening the central skeleton parts of square convolution kernels.
BibTeX
@inproceedings{ding2019acnet,
  author = {Xiaohan Ding and Yuchen Guo and Guiguang Ding and Jungong Han},
  title = {{ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00200},
}
Figure