Selecting Relevant Features from a Multi-domain Representation for Few-Shot Classification

Nikita Dvornik, Cordelia Schmid, Julien MairalECCV 2020 · OpenReview

Abstract

Popular approaches for few-shot classification consist of first learning a generic data representation based on a large annotated dataset, before adapting the representation to new classes given only a few labeled samples. In this work, we propose a new strategy based on feature selection, which is both simpler and more effective than previous feature adaptation approaches. First, we obtain a multi-domain representation by training a set of semantically different feature extractors. Then, given a few-shot learning task, we use our multi-domain feature bank to automatically select the most relevant representations. We show that a simple non-parametric classifier built on top of such features produces high accuracy and generalizes to domains never seen during training, leading to state-of-the-art results on MetaDataset and improved accuracy on mini-ImageNet.

Introduction

Few-shot classification (FSL) is the challenge of recognizing new categories from just one or five examples. While standard deep learning requires thousands of labeled images, FSL attempts to mimic human-like efficiency. However, it faces a significant hurdle: overfitting. When provided with only a handful of images, fine-tuning a massive neural network often results in the model memorizing specific pixels rather than learning generalizable concepts.

Existing solutions often rely on "meta-learning" to train complex adaptation modules. While innovative, these modules frequently fail when the test task comes from a domain (e.g., satellite imagery) significantly different from the training data (e.g., consumer photos). This paper introduces SUR (Selecting from Universal Representations), a strategy that shifts the focus from complex model adaptation to intelligent feature selection.

Methods: From Adaptation to Selection

The core philosophy of SUR is that choosing from a fixed set of high-quality, diverse features is more robust than trying to transform or "adapt" features when data is scarce. The process occurs in two distinct phases:

1. Building a Multi-domain Representation

The researchers first create a "feature bank" by training diverse feature extractors. They explore two primary architectures for this:

  • Independent Networks: Training separate ResNets on different datasets (e.g., birds, textures, characters).
  • Parametric Families (SUR-pf): Using a single "universal" ResNet and adding tiny, domain-specific FiLM layers (Feature-wise Linear Modulation). This approach is highly efficient, adding only 0.5% more parameters than a standard network while maintaining diverse representations.
The SUR pipeline: (Left) Pre-training a bank of diverse feature extractors. (Right) Selecting relevant features for a specific few-shot task using a support set.

2. Feature Selection and Classification

When a new few-shot task arrives, the model concatenates all features from the bank. It assigns a weight ($\lambda$) to each feature block. Using the few labeled examples provided (the "support set"), the model performs a few steps of gradient descent to optimize these weights. During this optimization, weights naturally tend toward 0 or 1, effectively "switching on" relevant experts (e.g., bird features for a bird task) and "switching off" irrelevant ones. Finally, a simple Nearest Centroid Classifier (NCC) is built on top of these weighted features.

Comparison of feature bank architectures: (a) Independent networks for each domain vs. (b) a parametric family sharing weights with domain-specific layers.

Evaluation and Results

The authors evaluated SUR on Meta-Dataset, a massive benchmark testing cross-domain generalization across 13 diverse datasets, including natural images (ImageNet, CU-Birds), specialized domains (Aircraft, Textures), and symbolic data (Omniglot, MNIST).

Key Findings

  • State-of-the-Art Performance: SUR outperformed previous leading methods like CNAPs on 9 out of 13 datasets in the Meta-Dataset benchmark.
  • Superior Generalization: The strongest gains were observed on "unseen" domains—datasets the model never encountered during initial training (such as MNIST and CIFAR).
  • Intra-domain Benefits: On the standard mini-ImageNet (5-shot), SUR improved accuracy from 76.28% to 79.25% by treating different layers of the same network as a multi-domain bank.
  • Efficiency: The SUR-pf variant achieved results comparable to using eight separate networks while being significantly lighter in terms of memory and computation.

Conclusion

SUR demonstrates that feature selection is a powerful alternative to feature adaptation in few-shot learning. By leveraging a diverse bank of pre-trained representations and a simple masking mechanism, the model achieves robust generalization across widely varying domains. While the method introduces a small computational overhead at inference time due to weight optimization, its ability to handle unseen domains makes it a significant step forward for practical few-shot applications.

Summary updated

Abstract

Popular approaches for few-shot classification consist of first learning a generic data representation based on a large annotated dataset, before adapting the representation to new classes given only a few labeled samples. In this work, we propose a new strategy based on feature selection, which is both simpler and more effective than previous feature adaptation approaches. First, we obtain a multi-domain representation by training a set of semantically different feature extractors. Then, given a few-shot learning task, we use our multi-domain feature bank to automatically select the most relevant representations. We show that a simple non-parametric classifier built on top of such features produces high accuracy and generalizes to domains never seen during training, leading to state-of-the-art results on MetaDataset and improved accuracy on mini-ImageNet.
BibTeX
@inproceedings{dvornik2020selecting,
  author = {Nikita Dvornik and Cordelia Schmid and Julien Mairal},
  title = {{Selecting Relevant Features from a Multi-domain Representation for Few-Shot Classification}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=Y1Ed4LAbWY},
}
Figure