TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning

Xin Wang, Fisher Yu, Ruth Wang, Trevor Darrell, Joseph E. Gonzalez

Abstract

In low-shot learning, the scarcity of data often necessitates the use of generic, pre-trained feature embeddings. However, these static representations are frequently sub-optimal for specific downstream tasks. We introduce TAFE-Net (Task-Aware Feature Embedding Network), a framework that dynamically adapts a neural network's weights based on a task description. By utilizing a meta-learner to generate task-specific parameters through weight factorization, TAFE-Net produces feature embeddings tailored to the current objective. Our approach achieves significant performance gains—up to 15 points—on complex visual composition and zero-shot learning benchmarks, demonstrating the power of task-conditional feature extraction.

Introduction

In computer vision, images are typically represented as feature embeddings—mathematical summaries that capture essential visual information. While effective for general tasks, these "one-size-fits-all" embeddings struggle in low-shot learning scenarios where data for new categories is scarce. In such cases, fine-tuning a massive model is often infeasible, forcing practitioners to rely on generic features learned from massive datasets like ImageNet.

The core limitation is that the relevance of a feature depends on the task at hand. For instance, distinguishing a "vintage car" from a "modern car" requires attention to texture and style, whereas identifying a "red car" prioritizes color. TAFE-Net addresses this by "morphing" generic features into task-specific ones on the fly, without requiring extensive retraining for every new concept.

Conceptual overview of task-aware embeddings where the feature space is dynamically adjusted based on the target category (e.g., Dog vs. Cat).

Methodology: The TAFE-Net Architecture

TAFE-Net moves away from static feature extraction by introducing a dynamic, task-conditional pipeline. The architecture consists of two primary components:

  1. The Meta-Learner ($\\mathcal{G}$): This network acts as a "teacher." It receives a task description—such as a semantic word embedding (e.g., "Cat") or a small set of support images—and generates the parameters (weights) required by the prediction network to specialize for that task.
  2. The Prediction Network ($\\mathcal{F}$): This "student" network processes input images. Unlike standard networks with fixed weights, its layers are parameterized by the outputs of the meta-learner, resulting in Task-Aware Feature Embeddings (TAFE).

Technical Innovations

To make this dynamic parameter generation practical, the authors introduced several key mechanisms:

  • Weight Factorization: Generating millions of weights for a deep neural network is computationally prohibitive. TAFE-Net solves this by decomposing weights into a large set of static shared weights (learned during training) and a small set of task-specific weights generated by the meta-learner. This ensures that the model remains efficient while still allowing for significant adaptation.
  • Secondary Embedding Loss: To stabilize training, especially when tasks are sparse, a metric learning loss is applied. This forces the generated task embeddings and the resulting image embeddings into a joint space where semantically related concepts are clustered together.
  • Task-Independent Classifier: Because the feature extraction process itself is now "aware" of the task, the final classification stage is simplified. A universal binary classifier can be used to determine if the task-aware features match the target concept.
The TAFE-Net architecture showing the Meta-Learner generating task-specific parameters for the Prediction Network via weight factorization.

Experimental Evaluation

The authors evaluated TAFE-Net across a diverse range of benchmarks, including Zero-Shot Learning (ZSL) datasets like SUN, CUB, and AWA2, as well as Attribute-Object Composition tasks (MITStates).

Key Results

TAFE-Net demonstrated superior performance across all categories:

  • Visual Composition: On unseen attribute-object pairs (e.g., recognizing a "red bus" after only seeing "red" and "bus" separately), the model improved accuracy by 4 to 15 points over previous state-of-the-art methods.
  • Generalized Zero-Shot Learning (GZSL): TAFE-Net significantly outperformed other discriminative models, achieving a 16-point gain on AWA1 and a 17-point gain on aPY.
  • Robustness: The meta-learner proved robust to various task encoding formats (e.g., Word2Vec vs. One-hot), indicating it successfully extracts deep semantic meaning rather than just memorizing labels.

Conclusion and Future Work

The success of TAFE-Net highlights that feature extraction should not be a fixed pipeline. By making network layers conditional on the task, models can dynamically "attend" to relevant visual cues. While the current implementation relies on a frozen backbone, future work could explore end-to-end training or combining TAFE-Net's discriminative approach with generative models to further push performance boundaries in low-data regimes.

Summary updated

Abstract

Learning good feature embeddings for images often requires substantial training data. As a consequence, in settings where training data is limited (e.g., few-shot and zero-shot learning), we are typically forced to use a generic feature embedding across various tasks. Ideally, we want to construct feature embeddings that are tuned for the given task. In this work, we propose Task-Aware Feature Embedding Networks (TAFE-Nets) to learn how to adapt the image representation to a new task in a meta learning fashion. Our network is composed of a meta learner and a prediction network. Based on a task input, the meta learner generates parameters for the feature layers in the prediction network so that the feature embedding can be accurately adjusted for that task. We show that TAFE-Net is highly effective in generalizing to new tasks or concepts and evaluate the TAFE-Net on a range of benchmarks in zero-shot and few-shot learning. Our model matches or exceeds the state-of-the-art on all tasks. In particular, our approach improves the prediction accuracy of unseen attribute-object pairs by 4 to 15 points on the challenging visual attribute-object composition task.
BibTeX
@inproceedings{wang2019tafe,
  author = {Xin Wang and Fisher Yu and Ruth Wang and Trevor Darrell and Joseph E. Gonzalez},
  title = {{TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.00193},
}
Figure