Weakly Supervised Attention Pyramid Convolutional Neural Network for Fine-Grained Visual Classification

Yifeng Ding, Shaoguo Wen, Jiyang Xie, Dongliang Chang, Zhanyu Ma, Zhongwei Si, Haibin LingarXiv 2020 · DOI · OpenReview

Abstract

Classifying the sub-categories of an object from the same super-category (e.g. bird species, car and aircraft models) in fine-grained visual classification (FGVC) highly relies on discriminative feature representation and accurate region localization. Existing approaches mainly focus on distilling information from high-level features. In this paper, however, we show that by integrating low-level information (e.g. color, edge junctions, texture patterns), performance can be improved with enhanced feature representation and accurately located discriminative regions. Our solution, named Attention Pyramid Convolutional Neural Network (AP-CNN), consists of a) a pyramidal hierarchy structure with a top-down feature pathway and a bottom-up attention pathway, and hence learns both high-level semantic and low-level detailed feature representation, and b) an ROI guided refinement strategy with ROI guided dropblock and ROI guided zoom-in, which refines features with discriminative local regions enhanced and background noises eliminated. The proposed AP-CNN can be trained end-to-end, without the need of additional bounding box/part annotations. Extensive experiments on three commonly used FGVC datasets (CUB-200-2011, Stanford Cars, and FGVC-Aircraft) demonstrate that our approach can achieve state-of-the-art performance.

Introduction

Fine-grained visual classification (FGVC) involves distinguishing between highly similar sub-categories, such as specific species of birds or variants of aircraft. This task is significantly more challenging than general classification due to high intra-class variance (different poses or lighting for the same species) and low inter-class variance (different species sharing many common traits).

While most deep learning models emphasize high-level semantic features, the "devil is in the details" for FGVC. Crucial low-level information—such as specific textures, colors, or edge patterns—is often lost as networks go deeper. Furthermore, many existing models rely on expensive human-annotated bounding boxes to identify discriminative parts. AP-CNN addresses these challenges by learning to integrate multi-level features and localize key regions in a weakly supervised manner.

Methodology

The Attention Pyramid Model

The core of the AP-CNN is a dual-pathway architecture built upon a standard Feature Pyramid Network (FPN). It facilitates the exchange of information across different scales of the network.

Architecture of the proposed AP-CNN, featuring the Feature Pyramid, Attention Pyramid, and ROI Pyramid.
  • Top-down Feature Pathway: This pathway passes high-level semantic information down to lower layers, providing global context that helps in locating the object of interest.
  • Bottom-up Attention Pathway: This novel component uses Spatial Attention to identify where important regions are and Channel Attention to determine which features are most relevant. By passing detailed, low-level information back up to higher layers, the model ensures that the final classification is informed by both coarse shapes and fine-grained textures (e.g., the sharp pattern of a wing tip).

ROI Guided Refinement

To further enhance the model's robustness and focus, AP-CNN employs an ROI-guided refinement strategy during training:

  • ROI Guided Dropblock: Instead of standard random dropout, the model identifies the most discriminative region and intentionally "erases" it. This forces the network to discover alternative discriminative features (e.g., looking at the tail if the beak is hidden), leading to a more holistic understanding of the object.
  • ROI Guided Zoom-in: The model automatically identifies, crops, and scales the most informative regions. This "refined" view is processed in a second stage, allowing the network to focus its capacity on the most relevant details.

Experimental Results and Insights

The AP-CNN was evaluated on three benchmark datasets: CUB-200-2011 (birds), Stanford Cars, and FGVC-Aircraft. The model achieved state-of-the-art (SOTA) or near-SOTA performance:

  • CUB-200-2011: 88.4% accuracy (ResNet50), a significant improvement over the 84.1% baseline.
  • Stanford Cars: 95.4% accuracy, outperforming previous leaders like DCL.
  • FGVC-Aircraft: 94.1% accuracy.

Key Insights and Visualizations

Qualitative analysis through heatmaps confirms that the Attention Pyramid effectively captures multi-scale information. High-level layers focus on the overall object structure, while low-level layers pinpoint specific textures and details.

Visualizing the Attention Pyramid and ROI Pyramid for Marsh Wren and Rock Wren samples.

The "Hide-and-Seek" strategy implemented via ROI Guided Dropblock makes the model a more robust "detective," preventing over-reliance on a single feature. Crucially, the model achieves these results under weak supervision, learning to localize parts purely from category labels without manual bounding box annotations.

Conclusion

AP-CNN demonstrates that integrating low-level details with high-level semantics is vital for fine-grained classification. By utilizing a dual-pathway pyramid and an ROI-guided refinement loop, the model achieves superior accuracy and robustness. While there is a trade-off between computational cost and the depth of refinement, the approach offers a powerful, weakly supervised framework for tasks where local detail is paramount. Future work may extend this pyramidal attention mechanism to fine-grained segmentation and instance detection.

Summary updated

Abstract

Classifying the sub-categories of an object from the same super-category (e.g. bird species, car and aircraft models) in fine-grained visual classification (FGVC) highly relies on discriminative feature representation and accurate region localization. Existing approaches mainly focus on distilling information from high-level features. In this paper, however, we show that by integrating low-level information (e.g. color, edge junctions, texture patterns), performance can be improved with enhanced feature representation and accurately located discriminative regions. Our solution, named Attention Pyramid Convolutional Neural Network (AP-CNN), consists of a) a pyramidal hierarchy structure with a top-down feature pathway and a bottom-up attention pathway, and hence learns both high-level semantic and low-level detailed feature representation, and b) an ROI guided refinement strategy with ROI guided dropblock and ROI guided zoom-in, which refines features with discriminative local regions enhanced and background noises eliminated. The proposed AP-CNN can be trained end-to-end, without the need of additional bounding box/part annotations. Extensive experiments on three commonly used FGVC datasets (CUB-200-2011, Stanford Cars, and FGVC-Aircraft) demonstrate that our approach can achieve state-of-the-art performance. Code available at \url{http://dwz1.cc/ci8so8a}
BibTeX
@misc{ding2020weakly,
  author = {Yifeng Ding and Shaoguo Wen and Jiyang Xie and Dongliang Chang and Zhanyu Ma and Zhongwei Si and Haibin Ling},
  title = {{Weakly Supervised Attention Pyramid Convolutional Neural Network for Fine-Grained Visual Classification}},
  year = {2020},
  eprint = {2002.03353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.03353},
}
Figure