Masked Feature Generation Network for Few-Shot Learning

Yunlong Yu, Dingyi Zhang, Zhong JiIJCAI 2022 · DOI

TL;DR

This paper introduces the Masked Feature Generation Network (MFGN), a novel approach to Few-Shot Learning (FSL) that tackles the challenge of recognizing new classes with very few examples. MFGN uses an encoder-decoder architecture inspired by masked autoencoders to generate diverse and discriminative feature embeddings for novel classes, significantly improving classification accuracy on various benchmarks.

What Problem Does This Paper Address?

Few-Shot Learning (FSL) is a critical area in machine learning that aims to enable models to learn from a very limited number of examples for new, unseen classes. This is particularly challenging because deep learning models typically require vast amounts of data to generalize effectively. Existing FSL methods often focus on modeling intra-class distributions or learning discriminative embedding spaces. However, many data augmentation techniques struggle with mode collapse, failing to capture the true intra-class variations. This paper addresses the need for a method that can effectively generate richer, more diverse feature representations for novel classes, even when only a handful of examples are available, thereby improving generalization capabilities.

Methods

The core of MFGN is an encoder-decoder architecture, inspired by the success of Masked Autoencoders (MAE) in self-supervised learning.

  • Encoder: This component acts as a feature extractor. It takes available visual instances from a class and generates their feature embeddings.
  • Decoder: This component is a transformer-based generator. It takes the feature embeddings from the available instances, along with special "mask tokens" and "sample number embeddings," to reconstruct the feature embeddings of masked (i.e., unavailable) visual instances.
  • Masking Strategy: Similar to MAE, MFGN employs a high masking ratio. Only one visible feature representation is fed to the decoder to reconstruct the features of the masked instances. This forces the decoder to learn more meaningful patterns.
  • Loss Functions: MFGN is trained using a combination of three loss functions:
  • Reconstruction Loss (Lmse): A Mean Squared Error (MSE) loss that measures the difference between the predicted feature embeddings and the actual feature embeddings of the masked instances.
  • Classification Loss (Lce): A standard cross-entropy loss that encourages the encoder to learn discriminative features for classification.
  • Regularization Loss (Lreg): This loss ensures that the reconstructed visual features are also discriminative by regularizing the classifier's output (logits) of the reconstructed features to be close to those of the original, real feature embeddings.

The entire framework is trained end-to-end from scratch.

Datasets

The paper evaluates MFGN on four standard FSL benchmarks:

  • Fine-grained:
    • Caltech UCSD Birds (CUB)
    • StanfordDogs
  • Coarse-grained:
    • miniImageNet
    • CIFAR-FS

These datasets represent a variety of image categories and levels of detail, allowing for a comprehensive evaluation.

Key Insights and Ideas

  • Transformer for Feature Reconstruction: The use of a transformer decoder is highlighted as crucial for reconstructing diverse visual features, leveraging its ability to handle positional information effectively.
  • Intra-Class Variation is Key: The paper strongly emphasizes that capturing intra-class variations is vital for robust FSL, and MFGN's generative approach directly addresses this.
  • Masked Feature Generation: Unlike MAE which reconstructs image patches, MFGN reconstructs feature embeddings, demonstrating the adaptability of the masked autoencoder paradigm to different representation spaces.
  • Discriminative and Diverse Features: The combination of reconstruction and classification losses, along with the regularization term, effectively pushes the model to generate features that are both distinct from other classes and varied within the same class.

Key Results

MFGN demonstrates strong performance across all tested datasets and architectures:

  • Fine-grained Datasets (CUB, StanfordDogs): MFGN achieved state-of-the-art or competitive results, outperforming existing methods by significant margins. For example, on CUB, it achieved 69.36% accuracy in the 5-Way 1-Shot setting with ConvNet4, surpassing the second-best by 3.07%.
  • Coarse-grained Datasets (miniImageNet, CIFAR-FS): MFGN also showed competitive results, often outperforming other feature augmentation and metric-learning-based approaches. For instance, on CIFAR-FS with ConvNet4, it achieved 66.1% accuracy, improving by 0.8% over the second-best.
  • Generalization: The experimental results suggest that MFGN's generalization ability is superior to most competitors, especially in scenarios with extremely limited data.
  • Impact of Loss Functions: Both the reconstruction loss (Lmse) and the regularization loss (Lreg) were found to bring significant improvements to classification performance.
  • Augmented Features: Using augmented visual features consistently improved performance compared to not using them, confirming the utility of the generated features.

Limitations and Future Work

  • Hyperparameter Tuning: The paper notes that the performance is sensitive to hyperparameters $\lambda$ and $\gamma$, which control the weights of the reconstruction and regularization losses, respectively. Optimal values were found to be around $\lambda=0.4$ and $\gamma=0.8$.
  • Number of Synthesized Features: While augmented features are beneficial, the study found that increasing the number of synthesized visual features beyond a certain point (e.g., 5) yielded diminishing returns.
  • Future Work: The authors suggest that fine-tuning techniques, if applied to MFGN, could potentially further improve performance, as MFGN is trained end-to-end from scratch without fine-tuning during inference. They also highlight the importance of intra-class feature variation for FSL and how their transformer-based method effectively captures it, suggesting further exploration in this direction.

Summary updated

BibTeX
@inproceedings{yu2022masked,
  author = {Yunlong Yu and Dingyi Zhang and Zhong Ji},
  title = {{Masked Feature Generation Network for Few-Shot Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/513},
}
Figure