TL;DR
This paper introduces Compositional Prototypical Networks (CPN) to improve few-shot learning by learning reusable "component prototypes" for attributes. These component prototypes are then combined to form class prototypes, which are adaptively fused with visual prototypes to achieve state-of-the-art performance, especially in challenging 1-shot scenarios.
What Problem Does This Paper Address?
Few-shot learning (FSL) aims to enable models to recognize new classes with very few examples, mimicking human learning. While pre-training feature extractors has shown promise, it doesn't fully capture how humans leverage prior knowledge by composing learned concepts. Humans can break down objects into components (e.g., bird features like beak shape, wing pattern) and use these to understand new instances. Existing FSL methods often treat classes holistically or rely on pre-trained features without explicitly learning and reusing these fine-grained, transferable "component" knowledge. This gap limits the ability of models to generalize effectively, especially when data is extremely scarce (1-shot).
Methods
The core innovation of CPN lies in learning component prototypes for predefined attributes during a pre-training stage.
- Pre-training: A feature extractor ($f_\theta$) and a set of learnable component prototypes ($\mathbf{r}_j$) are trained on a base dataset. Each component prototype corresponds to a specific attribute. A class prototype ($\mathbf{p}_c$) is then constructed as a weighted sum of these normalized component prototypes, where the weights are determined by the attribute scores of the class. This process is optimized using a standard cross-entropy loss for global classification.
- Meta-training (Episodic Training): In this stage, the learned component prototypes are reused. For each class in a few-shot task, a compositional prototype ($\mathbf{p}_{comp}$) is formed by combining the learned component prototypes based on the class's attribute scores. Simultaneously, a visual prototype ($\mathbf{p}_{vis}$) is computed as the mean feature vector of the support examples for that class.
- Adaptive Prototype Fusion: A learnable weight generator ($G$) is introduced to adaptively fuse the compositional and visual prototypes. It learns a weight coefficient ($\lambda$) that balances the contribution of $\mathbf{p}_{comp}$ and $\mathbf{p}_{vis}$ to form a final fused prototype ($\mathbf{p}_{fuse}$). This fusion is crucial for leveraging both semantic (compositional) and visual information, especially when visual data is limited.
- Classification: The fused prototype is then used with cosine similarity to classify query set examples.
Datasets
The paper evaluates CPN on two standard datasets with human-annotated attribute information:
- Caltech-UCSD-Birds 200-2011 (CUB): A fine-grained bird species dataset with 200 classes and 312 attributes.
- SUN Attribute Database (SUN): A scene recognition dataset with 717 classes and 102 attributes.
For both datasets, the authors use category-level attribute score vectors, calculating the mean attribute score vector for each class from available image-level annotations.
Key Insights and Ideas
- Decompositional Learning for FSL: The central idea is that learning and reusing fine-grained attribute components (prototypes) is more effective for few-shot generalization than relying solely on holistic class representations or general feature extractors.
- Transferable Component Prototypes: The learned component prototypes are shown to be transferable meta-knowledge, meaning they can be effectively reused across different classes and tasks.
- Adaptive Fusion is Key: Simply concatenating or naively combining semantic and visual information is less effective than adaptively fusing them. The learnable weight generator allows the model to dynamically decide how much to rely on the compositional (semantic) versus visual prototype, which is particularly beneficial in low-data regimes.
- 1-Shot Performance Boost: The compositional approach provides significant gains in the 5-way 1-shot setting, highlighting its strength when visual prototypes are less reliable due to extreme data scarcity.
Key Results
CPN achieves state-of-the-art results on both CUB and SUN datasets across 5-way 1-shot and 5-way 5-shot settings.
- CUB Dataset: CPN (using ResNet12) achieves 87.29% accuracy in the 5-way 1-shot setting and 92.54% in the 5-way 5-shot setting, outperforming previous methods.
- SUN Dataset: CPN (using Conv4) achieves 80.45% in 5-way 1-shot and 81.56% in 5-way 5-shot. With a stronger backbone (ResNet12), performance jumps to 88.07% and 89.21%, respectively, demonstrating the benefit of a powerful feature extractor.
- Ablation Studies:
- Learned component prototypes (LCP) significantly outperform randomly initialized ones (RICP), confirming their learned semantic value.
- Adaptive fusion of learned component prototypes and visual prototypes (Ours) consistently performs better than naive fusion or using either prototype alone.
- Further optimizing component prototypes during meta-training leads to improved performance.
Limitations and Future Work
- Attribute Availability: The method relies on predefined attribute annotations. The quality and completeness of these attributes can influence performance.
- Attribute Granularity: While CPN focuses on finer-grained attributes than some prior work, there might be even more granular components that could be learned.
- Future Work: The authors suggest exploring the use of more powerful feature extractors and investigating how to automatically discover relevant attributes or components, rather than relying on predefined ones. They also highlight the potential for this approach to inspire further research into feature reusability and compositional representations in few-shot learning.
Summary updated