Multimodal Prototypical Networks for Few-shot Learning

Frederik Pahde, Mihai Marian Puscas, Tassilo Klein, Moin Nabi

Abstract: Although providing exceptional results for many computer vision tasks, state-of-the-art deep learning algorithms catastrophically struggle in low data scenarios. However, if data in additional modalities exist (e.g. text) this can compensate for the lack of data and improve the classification results. To overcome this data scarcity, we design a cross-modal feature generation framework capable of enriching the low populated embedding space in few-shot scenarios, leveraging data from the auxiliary modality. Specifically, we train a generative model that maps text data into the visual feature space to obtain more reliable prototypes. This allows to exploit data from additional modalities (e.g. text) during training while the ultimate task at test time remains classification with exclusively visual data. We show that in such cases nearest neighbor classification is a viable approach and outperform state-of-the-art single-modal and multimodal few-shot learning methods on the CUB-200 and Oxford-102 datasets.

Introduction

Deep learning models are notoriously data-hungry, often requiring hundreds of examples per category to avoid overfitting. This poses a significant challenge in Few-shot Learning (FSL), where only a handful of samples are available. A common approach is the Prototypical Network, which represents a class by the average (prototype) of its available samples in a high-dimensional embedding space. However, in 1-shot or 2-shot scenarios, these prototypes are often "noisy" and fail to capture the true essence of a category, potentially focusing on background elements or specific lighting rather than discriminative features.

This paper addresses data scarcity by leveraging an auxiliary modality: text. By using textual descriptions to "hallucinate" additional visual features, we can create more stable and accurate class representatives, even when only visual data is available at test time.

Methodology: Cross-Modal Feature Generation

We propose a framework that enriches the visual embedding space using a generative model. The architecture consists of three primary components:

  1. Image Encoder: A pre-trained ResNet-18 that maps images into a visual embedding space.
  2. Text-Conditional GAN (tcGAN): This is the core innovation. Instead of generating high-resolution pixels, the Generator ($G_t$) learns to map text embeddings directly into the visual embedding space. This process is computationally efficient and more robust than full image generation.
  3. Multimodal Prototypes: To classify a new image, the system computes a "Multimodal Prototype" by taking a weighted average of the few available real visual features and a "cloud" of synthetic features generated from the class's text descriptions.
The proposed multimodal architecture featuring an image encoder and a text-conditional GAN (tcGAN) that generates synthetic visual features from text embeddings.

The GAN is trained with an auxiliary classification loss, forcing the generated features to be discriminative enough to identify the correct class. Crucially, at inference time, the model only requires the visual image, making it practical for real-world applications where text might be unavailable.

Experiments and Results

The method was evaluated on two fine-grained classification datasets: CUB-200-2011 (200 bird species) and Oxford-102 (102 flower categories). Both datasets include multiple natural language descriptions for each image.

Quantitative comparison of our multimodal approach against single-modal and zero-shot learning (ZSL) baselines on the CUB and Oxford-102 datasets.

As shown in the results table, the multimodal approach consistently outperforms state-of-the-art single-modal and multimodal baselines. In the 1-shot scenario on CUB-200, our method achieved 70.39% accuracy, a substantial improvement over the image-only baseline of 62.65%. Furthermore, the method demonstrated lower variance, indicating that the generated features make performance less dependent on the specific choice of the "shot" image.

Discussion and Key Insights

  • Text as a Semantic Anchor: Text descriptions often highlight discriminative features (e.g., "yellow beak"). Injecting these into the visual prototype helps the model ignore noise (like messy backgrounds) and focus on defining attributes.
  • Prototype Shift: The inclusion of text-generated features causes the prototype to shift from the location of the limited available images toward the true center of the class distribution, improving the neighborhood for nearest-neighbor classification.
  • Limitations: The method's success depends on the quality of textual descriptions. Poor or irrelevant text could potentially degrade performance. Future work could involve jointly optimizing image and text encoders end-to-end to further align the modalities.

Conclusion

Multimodal Prototypical Networks provide a robust solution to the data scarcity problem in few-shot learning. By using text to imagine what a category looks like beyond a single image, the model creates more reliable prototypes and achieves superior classification accuracy. This framework paves the way for incorporating other auxiliary modalities, such as audio or structured metadata, into few-shot learning systems.

Summary updated

Abstract

Although providing exceptional results for many computer vision tasks, state-of-the-art deep learning algorithms catastrophically struggle in low data scenarios. However, if data in additional modalities exist (e.g. text) this can compensate for the lack of data and improve the classification results. To overcome this data scarcity, we design a cross-modal feature generation framework capable of enriching the low populated embedding space in few-shot scenarios, leveraging data from the auxiliary modality. Specifically, we train a generative model that maps text data into the visual feature space to obtain more reliable prototypes. This allows to exploit data from additional modalities (e.g. text) during training while the ultimate task at test time remains classification with exclusively visual data. We show that in such cases nearest neighbor classification is a viable approach and outperform state-of-the-art single-modal and multimodal few-shot learning methods on the CUB-200 and Oxford-102 datasets.
BibTeX
@inproceedings{pahde2021multimodal,
  author = {Frederik Pahde and Mihai Marian Puscas and Tassilo Klein and Moin Nabi},
  title = {{Multimodal Prototypical Networks for Few-shot Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00269},
}
Figure