Grad-CAM guided channel-spatial attention module for fine-grained visual classification

Shuai Xu, Dongliang Chang, Jiyang Xie, Zhanyu Ma

Abstract

Fine-grained visual classification (FGVC) is becoming an important research field, due to its wide applications and the rapid development of computer vision technologies. The current state-of-the-art (SOTA) methods in the FGVC usually employ attention mechanisms to first capture the semantic parts and then discover their subtle differences between distinct classes. The channel-spatial attention mechanisms, which focus on the discriminative channels and regions simultaneously, have significantly improved the classification performance. However, the existing attention modules are poorly guided since part-based detectors in the FGVC depend on the network learning ability without the supervision of part annotations. As obtaining such part annotations is labor-intensive, some visual localization and explanation methods, such as gradient-weighted class activation mapping (Grad-CAM), can be utilized for supervising the attention mechanism. We propose a Grad-CAM guided channel-spatial attention module for the FGVC, which employs the Grad-CAM to supervise and constrain the attention weights by generating the coarse localization maps. To demonstrate the effectiveness of the proposed method, we conduct comprehensive experiments on three popular FGVC datasets, including CUB-200-2011, Stanford Cars, and FGVC-Aircraft datasets. The proposed method outperforms the SOTA attention modules in the FGVC task. In addition, visualizations of feature maps also demonstrate the superiority of the proposed method against the SOTA approaches.

Introduction

Fine-grained visual classification (FGVC) involves distinguishing between highly similar sub-categories, such as specific species of birds or models of aircraft. This task is significantly more challenging than general image classification due to the "tiny inter-class differences" and "significant intra-class diversity." For example, two different bird species may only differ by a small patch of color on their wings, while the same bird can look vastly different depending on its pose or lighting conditions.

Modern deep learning approaches typically use attention mechanisms to localize these small, discriminative features. However, these modules are often unsupervised, meaning they must discover relevant parts without explicit guidance. Traditionally, researchers addressed this by using manual part annotations (e.g., bounding boxes around a bird's head), but this process is prohibitively labor-intensive. This paper introduces a novel method to provide "guidance" to attention modules by repurposing Grad-CAM—an interpretability tool—as a supervisory signal during training.

Methodology

The proposed architecture centers on a Grad-CAM guided channel-spatial attention module. This approach integrates three primary components to refine the model's focus without manual labels.

The proposed Grad-CAM guided channel-spatial attention module architecture.

Channel-Spatial Attention

The module identifies both which feature channels are informative (channel attention) and where those features are located in the spatial domain (spatial attention). By focusing on both simultaneously, the model can better capture complex semantic parts.

Grad-CAM as a Teacher

Gradient-weighted Class Activation Mapping (Grad-CAM) is typically used post-hoc to explain model decisions via heatmaps. In this work, Grad-CAM is utilized during the training phase. It calculates the importance of each feature channel based on the gradients flowing back from the predicted class, providing a "ground truth" of what the model should be attending to for a correct classification.

GGAM-Loss

The core innovation is the Grad-CAM Guided Attention Mechanism Loss (GGAM-Loss). This loss function uses a symmetrical Kullback-Leibler (KL) divergence to minimize the difference between the forward-pass attention weights and the backward-pass Grad-CAM importance maps. This creates a self-supervised feedback loop where the model's own internal explanations teach its attention mechanisms to be more precise and less prone to background noise.

Experiments and Results

The researchers evaluated the proposed method on three standard FGVC benchmarks: CUB-200-2011 (200 bird species), Stanford Cars (196 car models), and FGVC-Aircraft (100 aircraft types). The experiments utilized VGG19 and ResNet50 backbones without any manual bounding box or part annotations.

The method achieved state-of-the-art (SOTA) or near-SOTA results across all datasets:

  • CUB-200-2011: Achieved 88.45% accuracy, outperforming previous methods like ACNet (88.10%) and TASN (87.90%).
  • FGVC-Aircraft: Achieved 93.42% accuracy, a notable improvement over DCL (93.00%).
  • Stanford Cars: Achieved 94.41% accuracy. While slightly behind ACNet (94.60%), the proposed method is significantly less complex and more parameter-efficient.

Qualitative Analysis

Visualizations of the feature maps provide clear evidence of the module's effectiveness in refining the model's "gaze."

Visual comparison of attention maps with and without the proposed GGAM-Loss.

As illustrated in Figure 4, attention maps generated without GGAM-Loss are often scattered and include significant background interference. When the GGAM-Loss is applied, the attention maps tighten significantly around the most relevant parts of the object. The combination of channel-spatial attention with the proposed loss (bottom right) yields the most precise localization, effectively suppressing noise and focusing on discriminative features like the bird's head and body.

Conclusion

The Grad-CAM guided channel-spatial attention module provides a robust, self-supervised framework for improving FGVC performance. By aligning forward-pass attention with backward-pass gradient information, the model learns to identify discriminative parts more effectively than unsupervised alternatives. While future work may explore its application to Vision Transformers (ViTs) or seek even finer-grained guidance, this method represents a significant advancement in high-performance classification without the need for expensive manual annotations.

Summary updated

Abstract

Fine-grained visual classification (FGVC) is becoming an important research field, due to its wide applications and the rapid development of computer vision technologies. The current state-of-the-art (SOTA) methods in the FGVC usually employ attention mechanisms to first capture the semantic parts and then discover their subtle differences between distinct classes. The channel-spatial attention mechanisms, which focus on the discriminative channels and regions simultaneously, have significantly improved the classification performance. However, the existing attention modules are poorly guided since part-based detectors in the FGVC depend on the network learning ability without the supervision of part annotations. As obtaining such part annotations is labor-intensive, some visual localization and explanation methods, such as gradient-weighted class activation mapping (Grad-CAM), can be utilized for supervising the attention mechanism. We propose a Grad-CAM guided channel-spatial attention module for the FGVC, which employs the Grad-CAM to supervise and constrain the attention weights by generating the coarse localization maps. To demonstrate the effectiveness of the proposed method, we conduct comprehensive experiments on three popular FGVC datasets, including CUB-$200$-$2011$, Stanford Cars, and FGVC-Aircraft datasets. The proposed method outperforms the SOTA attention modules in the FGVC task. In addition, visualizations of feature maps also demonstrate the superiority of the proposed method against the SOTA approaches.
BibTeX
@inproceedings{xu2021grad,
  author = {Shuai Xu and Dongliang Chang and Jiyang Xie and Zhanyu Ma},
  title = {{Grad-CAM guided channel-spatial attention module for fine-grained visual classification}},
  booktitle = {MLSP},
  year = {2021},
  url = {https://openreview.net/forum?id=hDD2arvJ1q_},
}
Figure