Multimodal Prototypical Networks for Few-shot Learning

Frederik Pahde, Mihai Marian Puscas, Tassilo Klein, Moin Nabi

Introduction

Deep learning models typically require vast amounts of labeled data to achieve high performance. However, in many real-world scenarios, such as identifying rare biological species, only a handful of examples may be available. This challenge, known as few-shot learning (FSL), often results in "noisy" class representations when models rely solely on limited visual samples. Standard methods struggle because these few visual samples might be outliers that fail to generalize.

This paper introduces Multimodal Prototypical Networks, a framework designed to stabilize and enhance few-shot classification by incorporating auxiliary textual descriptions. By leveraging the descriptive power of language, the model can "hallucinate" additional visual features, creating more robust class prototypes that generalize better than those derived from images alone.

Methodology: Cross-Modal Feature Generation

The authors extend the standard Prototypical Networks architecture by introducing a cross-modal feature generation framework. Instead of relying only on the mean vector of available image embeddings (the visual prototype), the model uses a text-conditional Generative Adversarial Network (tcGAN) to augment the representation.

Architecture of the multimodal feature generation framework, showing the interaction between image/text encoders and the tcGAN.

The process involves several key steps:

  1. Feature Generation: A generator ($G_t$) takes a text embedding (from a description) and random noise to produce synthetic visual features in the same latent space as real images. This "features over pixels" approach is computationally efficient and directly optimizes the classification representation.
  2. Multimodal Prototypes: For a novel class, the model generates numerous synthetic features from text descriptions. The final "Multimodal Prototype" ($p^M$) is a weighted fusion of the real visual prototype and this generated textual prototype.
  3. Stability via Diversity: By varying the noise input to the GAN, the model generates a diverse set of features from a single description, effectively "filling in" the sparse embedding space and making classification boundaries more robust.
  4. Inference: At test time, the model performs a simple nearest-neighbor search. An unseen image is mapped to the embedding space, and the model assigns it to the class of the closest multimodal prototype.

Experimental Results

The approach was evaluated on two benchmark datasets for fine-grained classification: CUB-200-2011 (birds) and Oxford-102 (flowers). Each dataset provides images paired with multiple textual descriptions.

Quantitative results comparing the multimodal approach against image-only and zero-shot baselines across different shot counts.

As shown in the table above, the multimodal approach consistently outperforms standard baselines and previous state-of-the-art methods:

  • Significant Gains in 1-Shot Learning: On the CUB-200 dataset, the method achieved 70.39% accuracy, a substantial improvement over the 62.65% image-only baseline. Similar trends were observed for Oxford-102, reaching 86.52%.
  • Scaling with Textual Data: The researchers found that accuracy improves as the number of textual descriptions per image increases. Even adding a second description provides a significant boost, confirming that linguistic diversity helps the model triangulate the correct visual concept.

Discussion and Conclusion

The success of Multimodal Prototypical Networks highlights the value of using text as a "prior" to correct the "prototype shift" that occurs when visual data is scarce. By generating latent features rather than full images, the model achieves high efficiency and performance.

However, the method's performance is tied to the quality of textual descriptions. Future research may explore dynamic weighting of modalities—allowing the model to decide how much to trust text versus images based on data quality—and joint optimization of encoders to further align the visual and textual embedding spaces.

Summary updated

Abstract

Although providing exceptional results for many computer vision tasks, state-of-the-art deep learning algorithms catastrophically struggle in low data scenarios. However, if data in additional modalities exist (e.g. text) this can compensate for the lack of data and improve the classification results. To overcome this data scarcity, we design a cross-modal feature generation framework capable of enriching the low populated embedding space in few-shot scenarios, leveraging data from the auxiliary modality. Specifically, we train a generative model that maps text data into the visual feature space to obtain more reliable prototypes. This allows to exploit data from additional modalities (e.g. text) during training while the ultimate task at test time remains classification with exclusively visual data. We show that in such cases nearest neighbor classification is a viable approach and outperform state-of-the-art single-modal and multimodal few-shot learning methods on the CUB-200 and Oxford-102 datasets.
BibTeX
@inproceedings{pahde2021multimodal,
  author = {Frederik Pahde and Mihai Marian Puscas and Tassilo Klein and Moin Nabi},
  title = {{Multimodal Prototypical Networks for Few-shot Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00269},
}
Figure