Multi-view Feature Augmentation with Adaptive Class Activation Mapping

Xiang Gao, Yingjie Tian, Zhiquan QiIJCAI 2021 · DOI

TL;DR

This paper introduces Multi-view Feature Augmentation (MV-FeaAug), a novel module that enhances image classification by intelligently sampling and ensembling local features from different "views" of an image's feature maps. By using an adaptive class activation mapping technique (AdaCAM), the module focuses on discriminative regions, leading to improved robustness and accuracy without increasing model complexity significantly.

What Problem Does This Paper Address?

Traditional image classification models often rely on Global Average Pooling (GAP) to condense feature maps into a single vector for classification. While efficient, GAP can be problematic when the main object of interest occupies only a small portion of an image, as background features can dilute or corrupt the representation. This leads to less discriminative features, especially for images with significant scale variations in the target object. The paper addresses this by proposing a method to extract and leverage multiple local feature representations, making the model more robust to such variations and improving overall classification performance.

Methods

The core of the proposed method is the Multi-view Feature Augmentation (MV-FeaAug) module. It works in conjunction with a standard CNN backbone.

  1. Adaptive Class Activation Mapping (AdaCAM): This is a key innovation. Unlike traditional CAM or Grad-CAM, AdaCAM generates attention maps in a feed-forward pass without requiring the ground truth label. It achieves this by modifying the classifier head to include a 1x1 convolutional layer before GAP, allowing it to produce class-discriminative feature maps. These maps are then used to identify "anchor points" that highlight regions of interest.
  2. Multi-view Local Feature Sampling: Using the anchor points identified by AdaCAM, the MV-FeaAug module samples multiple local regions (of varying scales) from the CNN's final feature maps. These sampled regions are then pooled to create diverse, class-representative local feature vectors.
  3. Feature Augmentation and Ensemble: These sampled local features are treated as augmented samples. The final prediction is made by ensembling the predictions from each of these multi-view local features, rather than relying on a single global representation.

The MV-FeaAug module is designed to be plug-and-play, replacing the standard GAP layer.

Datasets

The authors evaluated their method on a variety of object, scene, and action classification datasets:

  • Caltech101
  • Imagenette
  • Corel5k
  • Scene15
  • MIT Indoor67
  • Stanford Action40
  • UIUC Event8
  • UCMLU
  • RSSCN7
  • AID

Key Insights and Ideas

  • Local Features Matter More Than Global: The paper strongly suggests that for robust image classification, focusing on discriminative local features is more effective than relying solely on a global average representation, especially when object scale varies.
  • Label-Independent Attention is Powerful: AdaCAM's ability to generate attention maps without needing the ground truth label is crucial. This makes it applicable during inference when the label is unknown and allows for more adaptive feature selection.
  • Feature Augmentation Outperforms Image Augmentation: The authors argue that augmenting features in the feature space (by sampling diverse local views) is more effective than traditional image-level augmentation techniques like Mixup or Cutout.
  • Efficiency and Simplicity: MV-FeaAug achieves significant performance gains with minimal added complexity, primarily an auxiliary classifier head with a single 1x1 convolutional layer.

Key Results

  • Consistent Performance Gains: Across various CNN backbones (VGG16, ResNet50, ResNeXt50, MobileNet) and numerous datasets, MV-FeaAug consistently outperformed the baseline GAP approach, achieving accuracy improvements of several percentage points. For example, on VGG16, MV-FeaAug achieved 76.05% on Caltech101 compared to 70.42% for GAP.
  • Competitive with Attention Modules: MV-FeaAug showed superior or comparable performance to established attention modules (SE, CBAM, GCNet, SKNet) while adding less model complexity.
  • Effective Localization: Visualizations of AdaCAM attention maps showed strong alignment with class-discriminative regions, similar to CAM, but with the advantage of being feed-forward and label-independent.
  • Impact of K: Increasing the number of sampled local features (K) generally led to improved validation accuracy, indicating the benefit of exploring more views.

Limitations and Future Work

  • Hyperparameter Tuning: The number of anchor points (K) and the region sizes (LR) are hyperparameters that require tuning. While the paper provides some guidance, optimal values might vary across datasets.
  • Computational Cost: While the module is efficient, sampling multiple local features and ensembling predictions does introduce some computational overhead compared to a simple GAP.
  • Future Work: The authors suggest exploring more complex classifier heads (beyond single-FC layers) and prototype-based classifiers, which they demonstrate can further benefit from MV-FeaAug. They also hint at investigating the impact of different loss functions for the auxiliary classifier.

Summary updated

BibTeX
@inproceedings{gao2021multi,
  author = {Xiang Gao and Yingjie Tian and Zhiquan Qi},
  title = {{Multi-view Feature Augmentation with Adaptive Class Activation Mapping}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/94},
}
Figure