Abstract
The capability of the self-attention mechanism to model long-range dependencies has catapulted its deployment in vision models. Unlike convolution operators, self-attention offers an infinite receptive field and enables compute-efficient modeling of global dependencies. However, existing state-of-the-art attention mechanisms incur high compute and/or parameter overheads, making them unfit for compact convolutional neural networks (CNNs). In this work, we propose a simple yet effective "Ultra-Lightweight Subspace Attention Mechanism" (ULSAM), which infers different attention maps for each feature map subspace. We argue that learning separate attention maps for each feature subspace enables multi-scale and multi-frequency feature representation, which is more desirable for fine-grained image classification. ULSAM is end-to-end trainable and can be deployed as a plug-and-play module in pre-existing compact CNNs. Notably, our work is the first attempt that uses a subspace attention mechanism to increase the efficiency of compact CNNs. We achieve $\approx$13% and $\approx$25% reduction in both the FLOPs and parameter counts of MobileNet-V2 with a 0.27% and more than 1% improvement in top-1 accuracy on the ImageNet-1K and fine-grained image classification datasets (respectively).
Introduction: The Efficiency-Performance Trade-off
In computer vision, there is a constant tug-of-war between model performance and efficiency. While self-attention mechanisms have revolutionized the field by allowing models to capture global context, they are often too computationally expensive for compact networks designed for mobile devices, such as MobileNet or ShuffleNet. Standard CNNs are inherently local; a $3 \times 3$ filter only sees a small patch of an image at a time. To capture global dependencies, models typically require deep stacking of layers, which increases weight and latency.
Existing attention modules like SE-Net or CBAM, while effective, still introduce overhead that can be prohibitive for real-time mobile applications. ULSAM addresses this by providing the benefits of global context without the typical "attention tax."
The ULSAM Architecture
The core innovation of ULSAM is the Subspace Attention Mechanism. Instead of calculating a single global attention score for all channels, the module divides feature maps into $g$ mutually exclusive subspaces.
As shown in the architecture diagram, the process involves:
- Subspace Partitioning: Dividing input feature maps into groups.
- Efficient Feature Gathering: Using a Depthwise (DW) $1 \times 1$ convolution followed by a $3 \times 3$ Max Pooling layer to capture local informative regions efficiently.
- Attention Map Generation: Applying a single Pointwise (PW) $1 \times 1$ convolution and a Softmax gating mechanism to produce a spatial attention map for that specific subspace.
- Feature Redistribution: Multiplying the map back into the subspace features with a residual connection to preserve information flow.
By using only a single $1 \times 1$ filter per subspace and avoiding heavy Multi-Layer Perceptrons (MLPs), ULSAM remains significantly leaner than its predecessors.
Efficiency and Performance
ULSAM exploits the linear relationship between feature map subspaces to reduce parameters. This approach allows the model to simultaneously attend to "low-frequency" features (general shapes) and "high-frequency" features (fine details), which is critical for fine-grained tasks like bird or dog breed classification.
Quantitatively, ULSAM outperforms existing modules in efficiency. As illustrated in the comparison table, ULSAM requires only 1,000 parameters for a standard feature volume, whereas the Non-local block requires 524,000 and BAM requires 84,000. On the ImageNet-1K benchmark, ULSAM reduced MobileNet-V2's FLOPs by ~13% and parameters by ~25% while improving Top-1 accuracy by 0.27%. Gains were even more pronounced on specialized datasets like CUB-200-2011 (Birds) and Stanford Dogs, where accuracy improved by over 1%.
Conclusion
ULSAM demonstrates that attention mechanisms do not have to be computationally expensive to be effective. By rethinking attention at the subspace level, it enables compact CNNs to achieve higher accuracy with a smaller footprint. This makes it an ideal candidate for resource-constrained edge devices where both performance and efficiency are paramount.
Summary updated