Flexible Few-Shot Learning with Contextual Similarity

Mengye Ren, Eleni Triantafillou, Kuan-Chieh Wang, James Lucas, Jake Snell, Xaq Pitkow, Andreas S. Tolias, Richard S. Zemel

Abstract: Despite impressive progress in deep learning, generalizing far beyond the training distribution is an important open challenge. In this work, we consider few-shot classification, and aim to shed light on what makes some novel classes easier to learn than others, and what types of learned representations generalize better. To this end, we define a new paradigm in terms of attributes — simple building blocks of which concepts are formed — as a means of quantifying the degree of relatedness of different concepts. Our empirical analysis reveals that supervised learning generalizes poorly to new attributes, but a combination of self-supervised pretraining with supervised finetuning leads to stronger generalization. The benefit of self-supervised pretraining and supervised finetuning is further investigated through controlled experiments using random splits of the attribute space, and we find that predictability of test attributes provides an informative estimate of a model's generalization ability.

Introduction

Deep learning has achieved remarkable success in few-shot learning (FSL), where models are tasked with learning new classes from just a few examples. However, traditional benchmarks often rely on test classes that are visually and semantically similar to training classes, which can mask failures in true out-of-distribution generalization. We currently lack a systematic way to quantify the "relatedness" between concepts, making it difficult to measure how well a model truly adapts to novelty.

This paper proposes that concepts are constructed from simpler, fundamental building blocks called "attributes" (e.g., a bird is defined by feathers, wings, and laying eggs). If a model is tested on a novel class that relies on attributes it was never trained to prioritize, it is likely to fail. To address this, we introduce Few-Shot Attribute Learning (FSAL), a task designed to test a model's ability to learn entirely novel, previously unlabeled attributes from a small support set.

The Few-Shot Attribute Learning (FSAL) Task

Unlike standard FSL, where an image belongs to one unambiguous class, FSAL is inherently multi-label. In this paradigm, an image's label changes depending on the "context" defined by the support set. The target attribute is the commonality among positive examples in the support set that distinguishes them from negative examples.

Examples of FSAL episodes from the Celeb-A dataset, where the target attribute determines the positive and negative samples.

This setup introduces a unique challenge: ambiguity. For instance, a support set containing smiling people with glasses could be targeting either "smiling" or "glasses." Resolving this ambiguity often requires more "shots" (examples) than traditional FSL tasks to correctly identify the intended visual context.

Methodology

We evaluate several representation learning strategies to determine which best supports generalization to novel attributes:

  1. Supervised Learning: Models are trained on absolute attribute IDs (Supervised Attributes, SA) or using episodic meta-learning techniques like ProtoNet and MAML.
  2. Unsupervised Learning: We use contrastive learning (SimCLR) to learn general-purpose features without any attribute labels, preserving a broader range of visual information.
  3. Hybrid Approaches: We combine the strengths of both by pretraining with SimCLR and then finetuning on training attributes using either episodic labels (UFTE) or absolute labels (UFTA).

At test time, simple classifiers such as Nearest Centroid (NC) or Logistic Regression (LR) are applied to these learned representations to solve novel FSAL episodes.

Datasets and Evaluation

Our study utilizes three datasets with rich attribute annotations to provide a diverse testing ground:

  • Celeb-A: Over 200,000 face images with 40 attributes (split into 14 training and 13 testing attributes).
  • Zappos-50K: Approximately 50,000 shoe images with 40 training and 39 testing attributes.
  • ImageNet-with-Attributes: A subset of ImageNet featuring 25 attributes (11 training, 10 testing).
FSAL task applied to the Zappos-50K dataset, demonstrating the framework's applicability to diverse visual domains.

Key Results and Insights

Our empirical analysis reveals several critical insights into model generalization:

  • Supervised Over-specialization: Standard supervised and meta-learning methods perform exceptionally well on training attributes but suffer a massive performance drop on novel test attributes. These models effectively "learn to ignore" features that are not relevant to the specific training tasks.
  • The Robustness of Self-Supervision: Unsupervised contrastive learning preserves a broader range of features, leading to significantly better generalization on novel attributes compared to purely supervised methods.
  • The Hybrid Advantage: The hybrid UFTA/UFTE methods provide the best overall performance. On Celeb-A (20-shot), UFTA achieved 84.14% accuracy, outperforming SA (78.86%) and ProtoNet (75.27%). On Zappos-50K, UFTE reached 92.20%.
  • Narrowing the Generalization Gap: While supervised models showed a training-to-test gap of ~9.4% on Celeb-A, the hybrid UFTA model narrowed this gap to just 1.4%.
  • Predicting Generalization: We introduced a Transferability Score (T-Score) based on the predictability of test attributes from training attributes. We found a strong correlation between this score and few-shot performance, providing a metric for estimating task difficulty.

Conclusion

Few-Shot Attribute Learning highlights a significant limitation in current supervised learning: the tendency to discard features that are not immediately task-relevant. By shifting the focus to attributes, we demonstrate that a combination of self-supervised pretraining and supervised finetuning creates more flexible representations capable of generalizing to novel visual concepts. Future work will explore extending this paradigm to more diverse, unlabeled domains and addressing the inherent ambiguity in human-labeled attributes.

Summary updated

Abstract

Despite impressive progress in deep learning, generalizing far beyond the training distribution is an important open challenge. In this work, we consider few-shot classification, and aim to shed light on what makes some novel classes easier to learn than others, and what types of learned representations generalize better. To this end, we define a new paradigm in terms of attributes -- simple building blocks of which concepts are formed -- as a means of quantifying the degree of relatedness of different concepts. Our empirical analysis reveals that supervised learning generalizes poorly to new attributes, but a combination of self-supervised pretraining with supervised finetuning leads to stronger generalization. The benefit of self-supervised pretraining and supervised finetuning is further investigated through controlled experiments using random splits of the attribute space, and we find that predictability of test attributes provides an informative estimate of a model's generalization ability.
BibTeX
@misc{ren2020flexible,
  author = {Mengye Ren and Eleni Triantafillou and Kuan-Chieh Wang and James Lucas and Jake Snell and Xaq Pitkow and Andreas S. Tolias and Richard S. Zemel},
  title = {{Flexible Few-Shot Learning with Contextual Similarity}},
  year = {2020},
  eprint = {2012.05895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.05895},
}
Figure