Probing Few-Shot Generalization with Attributes

Mengye Ren, Eleni Triantafillou, Kuan-Chieh Wang, James M. Lucas, Jake Snell, Xaq Pitkow, Andreas S. Tolias, Richard S. Zemel

Abstract

Despite impressive progress in deep learning, generalizing far beyond the training distribution is an important open challenge. In this work, we consider few-shot classification, and aim to shed light on what makes some novel classes easier to learn than others, and what types of learned representations generalize better. To this end, we define a new paradigm in terms of attributes -- simple building blocks of which concepts are formed -- as a means of quantifying the degree of relatedness of different concepts. Our empirical analysis reveals that supervised learning generalizes poorly to new attributes, but a combination of self-supervised pretraining with supervised finetuning leads to stronger generalization. The benefit of self-supervised pretraining and supervised finetuning is further investigated through controlled experiments using random splits of the attribute space, and we find that predictability of test attributes provides an informative estimate of a model's generalization ability.

Introduction

Deep learning models are notoriously "brittle" when they encounter data outside their training distribution. While Few-Shot Learning (FSL) aims to address this by teaching models to recognize new classes from a few images, current FSL benchmarks often fall short of truly testing generalization. In many cases, "novel" classes at test time are merely new combinations of familiar attributes the model encountered during training. To address this, we propose shifting the focus from classes to attributes, introducing a more rigorous test of a model's ability to adapt to entirely novel concepts.

The Few-Shot Attribute Learning (FSAL) Paradigm

In the Few-Shot Attribute Learning (FSAL) paradigm, we challenge models to learn entirely novel visual attributes (e.g., "wearing eyeglasses" or "lace-up shoes") from just a few examples. Unlike standard FSL, where the model might see a new bird species but has already seen "wings" and "beaks," FSAL ensures that the test attributes are disjoint from the training set. This forces the model to develop representations that are not merely tuned to specific labels but are generally useful for understanding visual data.

Positive and negative examples for attributes 'Male & Smiling' and 'Cheekbones & Earrings' in the Celeb-A dataset.

Methodology

We evaluated several representation learning strategies to determine which best supports attribute-based generalization:

  1. Supervised Attributes (SA): Training the model to predict specific training attributes.
  2. Unsupervised (U): Utilizing SimCLR, a contrastive learning method that learns by recognizing different augmentations of the same image without using labels.
  3. Hybrid (UFTE/UFTA): A two-stage approach starting with unsupervised pretraining followed by supervised finetuning on training attributes.
  4. Meta-Learning: Standard FSL algorithms such as Prototypical Networks and MAML.

At test time, we employ a simple Logistic Regression (LR) classifier with L1 regularization. This setup allows the model to perform "feature selection," identifying which components of its learned representation are most relevant to the new attribute defined by the few-shot support set.

Datasets and Benchmarks

The study introduces three benchmarks adapted for attribute-based splitting:

  • Celeb-A: Over 200,000 face images split into 14 training and 13 test attributes.
  • Zappos-50K: 50,000 shoe images categorized by material, style, and type.
  • ImageNet-with-Attributes: A subset of ImageNet with 25 annotated attributes like "spotted" or "metallic."
Positive and negative examples for the 'Women & Lace Up' attribute in the Zappos-50K dataset.

Key Insights and Results

Our empirical analysis yielded several critical findings regarding generalization:

  • Supervised Learning is "Greedy": Purely supervised models tend to discard information not strictly necessary for the training task. If an attribute like "eyeglasses" is not a training label, the model learns to ignore it, leading to failure when that attribute becomes the target at test time.
  • Self-Supervision as a "Feature Hoarder": Unsupervised contrastive learning (SimCLR) preserves a wider array of features because it lacks a specific label-driven objective. This "hoarding" behavior proves to be a significant advantage for generalization to novel attributes.
  • The Power of Hybrid Representations: The hybrid methods (UFTA/UFTE) consistently outperformed standard meta-learning and purely supervised approaches. On Celeb-A (5-shot), the UFTA model achieved 78.98% accuracy, significantly surpassing ProtoNet (72.12%) and MAML (71.24%).
  • Predictability and the T-Score: We developed a Transferability Score (T-Score) to quantify generalization. We found that if a novel test attribute can be statistically predicted from the training attributes, the model generalizes significantly better.

Conclusion

By shifting the few-shot learning focus to attributes, we reveal the limitations of standard supervised learning in generalizing to novel concepts. Our results suggest that a combination of broad, unsupervised pretraining and targeted supervised finetuning creates the most robust "feature toolkit" for downstream tasks. Future work will explore more diverse unsupervised objectives and the automatic discovery of building-block attributes to further reduce reliance on human labeling.

Summary updated

Abstract

Despite impressive progress in deep learning, generalizing far beyond the training distribution is an important open challenge. In this work, we consider few-shot classification, and aim to shed light on what makes some novel classes easier to learn than others, and what types of learned representations generalize better. To this end, we define a new paradigm in terms of attributes -- simple building blocks of which concepts are formed -- as a means of quantifying the degree of relatedness of different concepts. Our empirical analysis reveals that supervised learning generalizes poorly to new attributes, but a combination of self-supervised pretraining with supervised finetuning leads to stronger generalization. The benefit of self-supervised pretraining and supervised finetuning is further investigated through controlled experiments using random splits of the attribute space, and we find that predictability of test attributes provides an informative estimate of a model's generalization ability.
BibTeX
@misc{ren2020probing,
  author = {Mengye Ren and Eleni Triantafillou and Kuan-Chieh Wang and James M. Lucas and Jake Snell and Xaq Pitkow and Andreas S. Tolias and Richard S. Zemel},
  title = {{Probing Few-Shot Generalization with Attributes}},
  year = {2020},
  eprint = {2012.05895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.05895},
}

Loading related content...

Figure