Learning Intact Features by Erasing-Inpainting for Few-shot Classification

Junjie Li, Zilei Wang, Xiaoming HuAAAI 2021 · OpenReview

Abstract

Few-shot classification aims to categorize the samples from unseen classes with only few labeled samples. To address such a challenge, many methods exploit a base set consisting of massive labeled samples to learn an instance embedding function, i.e., image feature extractor, and it is expected to possess good transferability among different tasks. Such characteristics of few-shot learning are essentially different from that of traditional image classification only pursuing to get discriminative image representations. In this paper, we propose to learn intact features by erasing-inpainting for few-shot classification. Specifically, we argue that extracting intact features of target objects is more transferable, and then propose a novel cross-set erasing-inpainting (CSEI) method. CSEI processes the images in the support set using erasing and inpainting, and then uses them to augment the query set of the same task. Consequently, the feature embedding produced by our proposed method can contain more complete information of target objects. In addition, we propose task-specific feature modulation to make the features adaptive to the current task. The extensive experiments on two widely used benchmarks well demonstrates the effectiveness of our proposed method, which can consistently get considerable performance gains for different baseline methods.

Introduction

In traditional deep learning, thousands of images are typically required to train a classifier. In Few-shot Classification (FSC), however, we only have a handful of labeled examples, known as the "support set." A major challenge identified by the authors is that standard models are often "lazy"—they tend to focus only on the most discriminative features (e.g., identifying a bird solely by its beak). While this works for massive datasets, it lacks the transferability needed for few-shot tasks where the support and query images might highlight different parts of the same object.

Methods: Learning "Intact" Features

The authors propose a two-pronged approach to force the model to see the "big picture" rather than just the most obvious parts.

Cross-Set Erasing-Inpainting (CSEI)

The core of the method is Cross-Set Erasing-Inpainting (CSEI). First, using Class Activation Maps (CAM), the model identifies the most discriminative region of a support image and erases it. Instead of leaving a black hole, which creates noisy edges, they use a pre-trained inpainting network (EdgeConnect) to fill the hole with plausible textures.

These modified images are then added to the query set during training. To correctly classify these "incomplete" queries, the feature extractor is forced to find other useful parts of the object (like legs or tails) that it previously ignored. This process ensures the feature embedding contains more complete, or "intact," information about the target objects.

Comparison of Class Activation Maps (CAMs) between baseline models and the proposed method. (a) Original images. (b-d) CAMs showing how the proposed method (d) captures more complete object information compared to baselines (b, c).

Task-Specific Feature Modulation (TSFM)

Since different tasks require different focus (e.g., distinguishing "husky vs. wolf" requires different features than "husky vs. car"), the authors introduce Task-Specific Feature Modulation (TSFM). This module uses channel-wise and spatial attention to adapt the extracted features to the specific set of classes in the current few-shot "episode."

Key Results and Insights

The proposed method was evaluated on two gold-standard benchmarks: miniImageNet and tieredImageNet.

  • Consistent Accuracy Boost: On miniImageNet (5-way 1-shot), adding these modules to the CAN baseline boosted accuracy from 64.42% to 67.59%. Combined with the EMD-9s model, they achieved a state-of-the-art 68.94% (1-shot) and 85.07% (5-shot).
  • Intact > Discriminative: The results confirm that for few-shot learning, a feature extractor that understands the "whole" object is more valuable than one that only finds the "easiest" part.
  • The Inpainting Advantage: Simply blacking out regions creates artificial edges that can confuse a network. Inpainting makes the "erased" image look natural, ensuring the model focuses on the remaining object features.

Conclusion

By shifting the focus from purely discriminative features to "intact" object representations, the CSEI method significantly improves the transferability of few-shot classifiers. While the initial generation of inpainted images adds computational overhead during training, the resulting performance gains across various baseline models demonstrate the robustness and effectiveness of this approach.

Summary updated

Abstract

Few-shot classification aims to categorize the samples from unseen classes with only few labeled samples. To address such a challenge, many methods exploit a base set consisting of massive labeled samples to learn an instance embedding function, i.e., image feature extractor, and it is expected to possess good transferability among different tasks. Such characteristics of few-shot learning are essentially different from that of traditional image classification only pursuing to get discriminative image representations. In this paper, we propose to learn intact features by erasing-inpainting for few-shot classification. Specifically, we argue that extracting intact features of target objects is more transferable, and then propose a novel cross-set erasing-inpainting (CSEI) method. CSEI processes the images in the support set using erasing and inpainting, and then uses them to augment the query set of the same task. Consequently, the feature embedding produced by our proposed method can contain more complete information of target objects. In addition, we propose task-specific feature modulation to make the features adaptive to the current task. The extensive experiments on two widely used benchmarks well demonstrates the effectiveness of our proposed method, which can consistently get considerable performance gains for different baseline methods.
BibTeX
@inproceedings{li2021learning,
  author = {Junjie Li and Zilei Wang and Xiaoming Hu},
  title = {{Learning Intact Features by Erasing-Inpainting for Few-shot Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=pvIiGSZePIa},
}
Figure