Few-Shot Attribute Learning.

Mengye Ren, Eleni Triantafillou, Kuan-Chieh Wang, James Lucas, Jake Snell, Xaq Pitkow, Andreas S. Tolias, Richard S. Zemel

Abstract

Semantic concepts are frequently defined by combinations of underlying attributes. As mappings from attributes to classes are often simple, attribute-based representations facilitate novel concept learning with zero or few examples. A significant limitation of existing attribute-based learning paradigms, such as zero-shot learning, is that the attributes are assumed to be known and fixed. In this work, we study the rapid learning of attributes that were not previously labeled. Compared to standard few-shot learning of semantic classes, in which novel classes may be defined by attributes that were relevant at training time, learning new attributes imposes a stiffer challenge. We found that supervised learning with training attributes does not generalize well to new test attributes, whereas self-supervised pre-training brings significant improvement. We further experimented with random splits of the attribute space and found that predictability of test attributes provides an informative estimate of a model's generalization ability.

Introduction

In the realm of computer vision, few-shot learning (FSL) has traditionally focused on recognizing novel object categories (e.g., a specific breed of dog) after seeing only a handful of examples. However, a critical question remains: is the model truly learning a new concept, or is it simply recombining features it has already mastered during its extensive pre-training?

This paper introduces Few-Shot Attribute Learning (FSAL), a benchmark designed to test a model's ability to learn entirely novel visual primitives—attributes like "wearing a hat" or "pointed heels"—that were never explicitly labeled during the initial training phase. Unlike standard FSL, where the "novel" class might share many attributes with known classes, FSAL requires the model to identify and utilize features it may have previously been incentivized to ignore.

The Few-Shot Attribute Learning Task

The core of FSAL involves a two-stage process. First, a model is trained on a set of "base" attributes. Then, at test time, it is presented with a "support set" consisting of a few positive and negative examples of a previously unseen "target" attribute. The model must then generalize this new concept to a query set of images.

Examples of FSAL tasks with positive and negative support sets for novel attributes.

As shown in the figure above, these tasks can range from facial features in the Celeb-A dataset to specific shoe characteristics in Zappos-50K. The challenge lies in the fact that the model's internal representation must be rich enough to capture these diverse features, even if they weren't relevant to the original training objective.

The "Information Destruction" Trap

A key finding of this research is that standard supervised learning often fails in the FSAL setting. When a model is trained solely to predict a fixed set of attributes, it becomes "too efficient." It learns to discard any visual information that doesn't help it minimize its training loss. For instance, if a model is trained only to recognize hair color, it may effectively "blind" itself to the presence of eyeglasses. This phenomenon, termed "information destruction," leaves the model unable to adapt when "eyeglasses" becomes the target attribute at test time.

Methodology: The Power of Hybrid Representations

To combat information destruction, the authors propose a hybrid learning framework:

  1. Self-Supervised Pre-training (U): Using techniques like SimCLR, the model learns to distinguish every image from every other image without any labels. This encourages the retention of a broad, diverse set of features.
  2. Supervised Fine-tuning (UFTA/UFTE): The model is then fine-tuned on the base training attributes. This "hybrid" approach aims to combine the discriminative power of supervised learning with the feature-rich safety net of self-supervision.
  3. Inference: At test time, a simple classifier (like Logistic Regression) is trained on the support set using the fixed features from the hybrid model.

Experimental Results and Insights

The researchers evaluated their approach on three datasets: Celeb-A (faces), Zappos-50K (shoes), and ImageNet-with-Attributes. The results consistently favored the hybrid approach. On the Celeb-A 20-shot task, the hybrid UFTA model achieved 84.14% accuracy, significantly outperforming the pure supervised baseline of 78.86%.

Furthermore, the authors introduced the Transferability Score (T-Score). By measuring the statistical predictability of test attributes from training attributes in the label space, they found a strong correlation with model performance. This suggests that the difficulty of a few-shot task is intrinsically linked to how "related" the new attribute is to the model's prior knowledge.

Conclusion

Few-Shot Attribute Learning highlights a significant limitation in current supervised learning paradigms: the tendency to discard potentially useful information. By integrating self-supervised pre-training, models can maintain a more flexible and comprehensive feature space, allowing them to learn novel semantic concepts from minimal data. This work paves the way for more robust AI systems capable of adapting to the "unknown unknowns" of real-world visual environments.

Summary updated

Abstract

Semantic concepts are frequently defined by combinations of underlying attributes. As mappings from attributes to classes are often simple, attribute-based representations facilitate novel concept learning with zero or few examples. A significant limitation of existing attribute-based learning paradigms, such as zero-shot learning, is that the attributes are assumed to be known and fixed. In this work we study the rapid learning of attributes that were not previously labeled. Compared to standard few-shot learning of semantic classes, in which novel classes may be defined by attributes that were relevant at training time, learning new attributes imposes a stiffer challenge. We found that supervised learning with training attributes does not generalize well to new test attributes, whereas self-supervised pre-training brings significant improvement. We further experimented with random splits of the attribute space and found that predictability of test attributes provides an informative estimate of a model's generalization ability.
BibTeX
@misc{ren2020few,
  author = {Mengye Ren and Eleni Triantafillou and Kuan-Chieh Wang and James Lucas and Jake Snell and Xaq Pitkow and Andreas S. Tolias and Richard S. Zemel},
  title = {{Few-Shot Attribute Learning.}},
  year = {2020},
  eprint = {2012.05895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.05895},
}

Loading related content...

Figure