Standard convolutional neural networks rely heavily on square kernels—typically $3 \times 3$ or $5 \times 5$—to capture spatial features. While effective, these filters are computationally expensive, as their parameter count grows quadratically with the kernel size. Asymmetric convolutional decomposition offers a solution by factorizing a single $N \times N$ square convolution into a sequence of two one-dimensional filters: an $N \times 1$ convolution followed by a $1 \times N$ convolution.
This architectural shift allows models to maintain the same effective receptive field while significantly reducing the number of parameters and operations required.
The Logic of Factorization
The intuition behind asymmetric decomposition is that spatial correlations can often be captured independently across horizontal and vertical axes. As Szegedy et al. argue in the development of Inception-v3, a $3 \times 3$ convolution can be replaced by a $3 \times 1$ and a $1 \times 3$ convolution without losing the ability to model the same spatial context.
The mathematical advantage is clear: a $3 \times 3$ kernel requires 9 parameters per channel, whereas the factorized version requires only $3 + 3 = 6$, a 33% reduction. For larger kernels, the savings are even more dramatic; Li et al. demonstrate that decomposing large kernels into asymmetric strips expands the receptive field while drastically lowering the computational cost compared to dense square filters.
Beyond simple parameter reduction, this decomposition introduces additional non-linearity. By placing an activation function between the two 1D layers, the network can learn more complex representations than a single square layer would allow, as noted by Tetteh et al..
Efficiency in Real-Time and Edge Applications
Because asymmetric decomposition slashes the memory footprint and FLOPs, it has become a cornerstone for deploying deep learning on resource-constrained hardware.
- Object Detection and Tracking: Mai Cheng shows that integrating asymmetric kernels into YOLOv3 reduces parameters by one-third while maintaining tracking accuracy in high-noise environments. Similarly, Rao et al. utilize this spatial separation in Light-Net to achieve a model size under 10MB.
- Semantic Segmentation: For real-time tasks like drone-based navigation, Lei Sun demonstrates that $1 \times N$ and $N \times 1$ pairs enable high-resolution segmentation at real-time speeds on edge devices. Mazzini and Schettini further prove this efficiency, achieving over 100 FPS on the Cityscapes dataset by leveraging factorized convolutions.
- Medical Imaging: In high-precision tasks like intervertebral disc segmentation, Dolz et al. use asymmetric kernels to reduce memory requirements during training, which is often a bottleneck for 3D medical volumes.
Structural Variations and Hybrid Approaches
The concept of asymmetry extends beyond simple kernel shape into how channels and dimensions are handled.
Channel-wise Asymmetry: Rather than applying the same operation to all features, some researchers propose heterogeneous processing. Bilecen and Ayazoglu introduce a method that splits feature channels into unequal groups, applying spatial convolutions to one subset and pointwise kernels to another. This focuses expensive spatial processing only where it is most needed.
Temporal Decomposition: In video analysis, the principle is applied to 3D convolutions. Luo and Yuille decompose 3D kernels into parallel spatial and temporal paths, limiting expensive temporal reasoning to a specific subset of channels to balance motion modeling with efficiency.
Robustness and Signal Processing: Interestingly, asymmetric kernels may offer benefits beyond speed. Suo et al. suggest that horizontal and vertical decomposition enhances a model's robustness to image rotations and flips. In the domain of radio signals, Doan et al. found that $1 \times 3$ and $3 \times 1$ filters are uniquely suited to capturing the grid-like and circular patterns found in signal constellation diagrams.
Hardware and Implementation Constraints
Despite the theoretical savings, asymmetric kernels face practical challenges in deployment. Wen et al. point out that many high-performance computing libraries (like cuDNN) are heavily optimized for square $3 \times 3$ kernels. Consequently, the reduction in theoretical FLOPs from using $1 \times 3$ and $3 \times 1$ filters does not always translate to a linear increase in wall-clock speed, as the hardware may struggle to process these non-standard dimensions efficiently.
Nevertheless, as a design pattern, asymmetric decomposition remains a vital tool for researchers seeking to push the boundaries of what deep networks can achieve on limited computational budgets.
Key Foundational Work
Rethinking the Inception Architecture for Computer Vision (2016) — contributed 2 concept elements to this direction.
Blind Deconvolution in Nonminimum Phase Systems Using Cascade Structure (2007) — contributed 1 concept elements to this direction.