Abstract
The capability of the self-attention mechanism to model long-range dependencies has catapulted its deployment in vision models. Unlike convolution operators, self-attention offers an infinite receptive field and enables compute-efficient modeling of global dependencies. However, existing state-of-the-art attention mechanisms incur high compute and parameter overheads, making them unfit for compact convolutional neural networks (CNNs). We propose the "Ultra-Lightweight Subspace Attention Mechanism" (ULSAM), which infers different attention maps for each feature map subspace. Learning separate attention maps for each feature subspace enables multi-scale and multi-frequency feature representation, which is desirable for fine-grained image classification. ULSAM is end-to-end trainable and can be deployed as a plug-and-play module. We achieve ≈13% and ≈25% reduction in both FLOPs and parameter counts of MobileNet-V2 with improved top-1 accuracy on ImageNet-1K and fine-grained classification datasets.
Introduction
While deep learning has revolutionized computer vision, the best-performing models are often computationally expensive and memory-heavy. To address this, compact architectures like MobileNets and ShuffleNets utilize depthwise separable convolutions to maintain efficiency. However, these models often struggle to capture long-range dependencies—the ability to understand spatial relationships across the entire image.
Self-attention mechanisms solve this but are typically too "heavy" for mobile devices, incurring massive overhead in parameters and floating-point operations (FLOPs). ULSAM addresses this gap by providing the benefits of global context without exceeding the computational budget of mobile-first models.
The Subspace Attention Mechanism
The core innovation of ULSAM is the Subspace Attention Mechanism. Instead of computing a single massive attention map for all channels or a separate one for every channel, ULSAM divides input feature maps into $g$ mutually exclusive groups (subspaces).
For each subspace, the module generates a unique attention map using a highly optimized sequence:
- 1x1 Depthwise Convolution: Captures local channel information.
- 3x3 Max Pooling: Highlights informative spatial regions.
- 1x1 Pointwise Convolution: Uses a single filter to collapse information into a spatial map.
- Softmax Gating: Normalizes the weights.
This map is multiplied back into the subspace features, followed by a residual connection to preserve original information. By avoiding heavy Multi-Layer Perceptrons (MLPs) used in modules like SE-Net, ULSAM remains significantly lighter.
Comparative Efficiency
A key strength of ULSAM is its efficiency relative to other popular attention modules. As shown in the table below, ULSAM achieves a dramatic reduction in parameters and FLOPs compared to established mechanisms like Non-local networks or CBAM.
Unlike modules that rely on parameter-heavy non-linear MLPs to learn channel dependencies, ULSAM exploits linear relationships between feature map subspaces. This strategic design allows it to scale efficiently, especially in deeper layers where spatial resolution is smaller and semantic information is richer.
Experimental Results
The researchers evaluated ULSAM on ImageNet-1K and several fine-grained classification datasets, including Food-101, Birds-200-2011, and Stanford Dogs. Fine-grained tasks are particularly challenging because they require capturing subtle, high-frequency details to distinguish between similar classes (e.g., specific bird species).
Key findings include:
- Efficiency Gains: Integration into MobileNet-V2 resulted in a ~13% reduction in FLOPs and parameters on ImageNet and a ~25% reduction on fine-grained datasets.
- Accuracy Boost: Despite the reduced size, ULSAM-enhanced MobileNet-V2 saw a 0.27% improvement in Top-1 accuracy on ImageNet and over 1% improvement on fine-grained tasks.
- Multi-Scale Representation: By learning different attention maps for different subspaces, the network can focus on various "frequencies" of data—some subspaces capture coarse shapes while others focus on fine textures.
Conclusion
ULSAM demonstrates that attention mechanisms do not have to be computationally prohibitive for compact CNNs. By leveraging subspace-specific attention maps, the module enhances feature representation for complex tasks like fine-grained classification while simultaneously reducing the model's footprint. Future work may explore more complex spatial attention and broader global relations while maintaining the "ultra-lightweight" profile.
Summary updated