Three-Dimension Attention Mechanism and Self-Supervised Pretext Task for Augmenting Few-Shot Learning

Yong Liang, Zetao Chen, Daoqian Lin, Junwen Tan, Zhenhao Yang, Jie Li, Xinhai LiIEEE Access 2023 · DOI

TL;DR

This paper introduces a novel few-shot learning (FSL) framework that combines a three-dimensional attention mechanism with a self-supervised rotation pretext task. By forcing the model to focus on semantically rich features and learn image structure through unlabelled rotations, the authors achieved state-of-the-art performance across several benchmark datasets, particularly in challenging fine-grained classification tasks.

What Problem Does This Paper Address?

Few-shot learning (FSL) aims to train models that can recognize new categories from just a handful of examples (e.g., 1 or 5 images). The core challenge is the "data scarcity" problem: with so few samples, deep neural networks are prone to overfitting and struggle to generalize.

The authors argue that standard image-level labels (e.g., "this is a dog") are often too coarse. They don't provide enough information about the internal features or spatial relationships within an image. Existing metric-based methods (which classify by measuring distances between images) often fail to capture the most representative features, leading to poor robustness when the model encounters slightly different variations of a class.

Methods

The researchers proposed a two-pronged approach integrated into a standard ResNet12 backbone:

  1. Three-Dimensional Attention Mechanism: Inspired by the human brain and the SimAM module, this mechanism calculates weights for neurons based on an "energy function." Unlike traditional attention that looks at channels (what is in the image) or spatial locations (where things are) separately, this 3D approach evaluates both simultaneously. This helps the model prioritize semantically informative features while suppressing noise.
  2. Self-Supervised Pretext Task (Rotation): To squeeze more information out of the limited data without needing more manual labels, the authors used self-supervised learning (SSL). During training, the model is given images rotated by 0°, 90°, 180°, or 270° and tasked with predicting the rotation angle. This "proxy task" forces the network to learn the structural and geometric properties of the objects, which significantly improves its ability to generalize to new, unseen categories.

The final training loss is a 50/50 split between the standard supervised classification loss and the self-supervised rotation loss.

Datasets

The model was evaluated on four widely recognized FSL benchmarks:

  • mini-ImageNet: A subset of the massive ImageNet dataset (100 classes).
  • CIFAR-FS: A few-shot version of the CIFAR-100 dataset.
  • FC100: Another CIFAR-100 derivative, but split by "superclasses" to make it more difficult.
  • CUB200: A fine-grained dataset consisting entirely of different bird species, requiring the model to distinguish between very subtle differences.

Key Insights and Ideas

  • The Power of "Free" Labels: The rotation task acts as a form of "regularization" that provides richer supervision signals. It turns out that teaching a model to know which way is "up" helps it understand the object itself much better.
  • Synergy is Key: Ablation studies showed that while both 3D attention and SSL help individually, their combination provides a significant boost. SSL, in particular, showed a higher individual impact than the attention module alone, suggesting that structural understanding is a major bottleneck in FSL.
  • Fine-Grained Excellence: The method performed exceptionally well on the CUB200 (birds) dataset. This suggests that the 3D attention mechanism is particularly adept at finding the tiny, specific features (like beak shape or wing patterns) necessary for fine-grained recognition.

Key Results

The proposed method achieved state-of-the-art (SOTA) results across the board:

  • CUB200: Achieved 77.3% accuracy in 1-shot and 90.88% in 5-shot scenarios, significantly outperforming previous benchmarks like ProtoNet and MAML.
  • mini-ImageNet: Reached 66.02% (1-shot) and 84.7% (5-shot).
  • CIFAR-FS: Reached 72.76% (1-shot) and 86.71% (5-shot).

In almost every category, the model outperformed established methods, proving that the combination of 3D attention and self-supervision creates a much more robust feature extractor.

Limitations and Future Work

While the results are impressive, the authors note that the role of self-supervised learning seems more dominant than the attention mechanism in the small-sample domain. They suggest that future research should explore other types of proxy tasks (beyond just rotation) to see if even richer semantic information can be extracted.

Additionally, while the 3D attention module is "parameter-free" (it doesn't add new learnable weights), the self-supervised component does add some computational overhead during the training phase, though the authors describe this as minor. Future work could look into more computationally efficient ways to integrate these "self-study" tasks.

Summary updated

Abstract

The main challenge of few-shot learning lies in the limited labeled sample of data. In addition, since image-level labels are usually not accurate in describing the features of images, it leads to difficulty for the model to have good generalization ability and robustness. This problem has not been well solved yet, and existing metric-based methods still have room for improvement. To address this issue, we propose a few-shot learning method based on a three-dimension attention mechanism and self-supervised learning. The attention module is used to extract more representative features by focusing on more semantically informative features through spatial and channel attention. Self-supervised learning mainly adopts a proxy task of rotation transformation, which increases semantic information without requiring additional manual labeling, and uses this information for training in combination with supervised learning loss function to improve model robustness. We have conducted extensive experiments on four popular few-shot datasets and achieved state-of-the-art performance in both 5-shot and 1-shot scenarios. Experiment results show that our work provides a novel and remarkable approach to few-shot learning.
BibTeX
@inproceedings{liang2023three,
  author = {Yong Liang and Zetao Chen and Daoqian Lin and Junwen Tan and Zhenhao Yang and Jie Li and Xinhai Li},
  title = {{Three-Dimension Attention Mechanism and Self-Supervised Pretext Task for Augmenting Few-Shot Learning}},
  booktitle = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3285721},
}
Figure