ULSAM: Ultra-Lightweight Subspace Attention Module for Compact Convolutional Neural Networks

Rajat Saini, Nandan Kumar Jha, Bedanta Das, Sparsh Mittal, C. Krishna Mohan

Abstract

The capability of the self-attention mechanism to model long-range dependencies has catapulted its deployment in vision models. Unlike convolution operators, self-attention offers an infinite receptive field and enables compute-efficient modeling of global dependencies. However, existing state-of-the-art attention mechanisms incur high compute and parameter overheads, making them unfit for compact convolutional neural networks (CNNs). We propose the "Ultra-Lightweight Subspace Attention Mechanism" (ULSAM), which infers different attention maps for each feature map subspace. Learning separate attention maps for each feature subspace enables multi-scale and multi-frequency feature representation, which is desirable for fine-grained image classification. ULSAM is end-to-end trainable and can be deployed as a plug-and-play module. We achieve ≈13% and ≈25% reduction in both FLOPs and parameter counts of MobileNet-V2 with improved top-1 accuracy on ImageNet-1K and fine-grained classification datasets.

Introduction

While deep learning has revolutionized computer vision, the best-performing models are often computationally expensive and memory-heavy. To address this, compact architectures like MobileNets and ShuffleNets utilize depthwise separable convolutions to maintain efficiency. However, these models often struggle to capture long-range dependencies—the ability to understand spatial relationships across the entire image.

Self-attention mechanisms solve this but are typically too "heavy" for mobile devices, incurring massive overhead in parameters and floating-point operations (FLOPs). ULSAM addresses this gap by providing the benefits of global context without exceeding the computational budget of mobile-first models.

The Subspace Attention Mechanism

The core innovation of ULSAM is the Subspace Attention Mechanism. Instead of computing a single massive attention map for all channels or a separate one for every channel, ULSAM divides input feature maps into $g$ mutually exclusive groups (subspaces).

Detailed architecture of the ULSAM module showing the lean attention generation process for each subspace.

For each subspace, the module generates a unique attention map using a highly optimized sequence:

  1. 1x1 Depthwise Convolution: Captures local channel information.
  2. 3x3 Max Pooling: Highlights informative spatial regions.
  3. 1x1 Pointwise Convolution: Uses a single filter to collapse information into a spatial map.
  4. Softmax Gating: Normalizes the weights.

This map is multiplied back into the subspace features, followed by a residual connection to preserve original information. By avoiding heavy Multi-Layer Perceptrons (MLPs) used in modules like SE-Net, ULSAM remains significantly lighter.

Comparative Efficiency

A key strength of ULSAM is its efficiency relative to other popular attention modules. As shown in the table below, ULSAM achieves a dramatic reduction in parameters and FLOPs compared to established mechanisms like Non-local networks or CBAM.

Comparison of ULSAM against other attention modules. ULSAM shows significantly lower parameter and FLOP counts.

Unlike modules that rely on parameter-heavy non-linear MLPs to learn channel dependencies, ULSAM exploits linear relationships between feature map subspaces. This strategic design allows it to scale efficiently, especially in deeper layers where spatial resolution is smaller and semantic information is richer.

Experimental Results

The researchers evaluated ULSAM on ImageNet-1K and several fine-grained classification datasets, including Food-101, Birds-200-2011, and Stanford Dogs. Fine-grained tasks are particularly challenging because they require capturing subtle, high-frequency details to distinguish between similar classes (e.g., specific bird species).

Key findings include:

  • Efficiency Gains: Integration into MobileNet-V2 resulted in a ~13% reduction in FLOPs and parameters on ImageNet and a ~25% reduction on fine-grained datasets.
  • Accuracy Boost: Despite the reduced size, ULSAM-enhanced MobileNet-V2 saw a 0.27% improvement in Top-1 accuracy on ImageNet and over 1% improvement on fine-grained tasks.
  • Multi-Scale Representation: By learning different attention maps for different subspaces, the network can focus on various "frequencies" of data—some subspaces capture coarse shapes while others focus on fine textures.

Conclusion

ULSAM demonstrates that attention mechanisms do not have to be computationally prohibitive for compact CNNs. By leveraging subspace-specific attention maps, the module enhances feature representation for complex tasks like fine-grained classification while simultaneously reducing the model's footprint. Future work may explore more complex spatial attention and broader global relations while maintaining the "ultra-lightweight" profile.

Summary updated

Abstract

The capability of the self-attention mechanism to model the long-range dependencies has catapulted its deployment in vision models. Unlike convolution operators, self-attention offers infinite receptive field and enables compute- efficient modeling of global dependencies. However, the existing state-of-the-art attention mechanisms incur high compute and/or parameter overheads, and hence unfit for compact convolutional neural networks (CNNs). In this work, we propose a simple yet effective "Ultra-Lightweight Subspace Attention Mechanism" (ULSAM), which infers different attention maps for each feature map subspace. We argue that leaning separate attention maps for each feature subspace enables multi-scale and multi-frequency feature representation, which is more desirable for fine-grained image classification. Our method of subspace attention is orthogonal and complementary to the existing state-of-the- arts attention mechanisms used in vision models. ULSAM is end-to-end trainable and can be deployed as a plug-and- play module in the pre-existing compact CNNs. Notably, our work is the first attempt that uses a subspace attention mechanism to increase the efficiency of compact CNNs. To show the efficacy of ULSAM, we perform experiments with MobileNet-V1 and MobileNet-V2 as backbone architectures on ImageNet-1K and three fine-grained image classification datasets. We achieve ≈13% and ≈25% reduction in both the FLOPs and parameter counts of MobileNet-V2 with a 0.27% and more than 1% improvement in top-1 accuracy on the ImageNet-1K and fine-grained image classification datasets (respectively). Code and trained models are available at https://github.com/Nandan91/ULSAM.
BibTeX
@inproceedings{saini2020ulsam,
  author = {Rajat Saini and Nandan Kumar Jha and Bedanta Das and Sparsh Mittal and C. Krishna Mohan},
  title = {{ULSAM: Ultra-Lightweight Subspace Attention Module for Compact Convolutional Neural Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=_8bqPsRA-s},
}
Figure