Abstract
Attribute based knowledge transfer has proven very successful in visual object analysis and learning previously unseen classes. However, the common approach learns and transfers attributes without taking into consideration the embedded structure between the categories in the source set. Such information provides important cues on the intra-attribute variations. We propose to capture these variations in a hierarchical model that expands the knowledge source with additional abstraction levels of attributes. We also provide a novel transfer approach that can choose the appropriate attributes to be shared with an unseen class. We evaluate our approach on three public datasets: aPascal, Animals with Attributes and CUB-200-2011 Birds. The experiments demonstrate the effectiveness of our model with significant improvement over state-of-the-art.
Introduction: The Challenge of Intra-Attribute Variance
Zero-shot learning (ZSL) aims to classify objects the model has never seen during training by relying on semantic descriptions, usually in the form of attributes like "furry," "has a beak," or "lives in water." Historically, these attributes are learned globally: a model looks at all training images with a "beak" and learns a single, generalized classifier.
However, this approach suffers from intra-attribute variance. As shown in the figure below, a hummingbird's beak and an albatross's beak look entirely different. By forcing a model to learn a single global representation, the fine-grained, discriminative properties of the attribute are lost. This paper addresses this by asking: instead of transferring a generic "beak" concept to a novel bird, can we transfer more specific concepts by exploiting the taxonomic relationships between categories?
The Hierarchical Attribute Transfer (HAT) Model
The authors propose the Hierarchical Attribute Transfer (HAT) model, which moves away from flat attribute lists and instead embeds classes into a tree structure (using the WordNet ontology). The framework operates through three primary mechanisms:
- Attribute Propagation: The model propagates attribute labels bottom-up. If leaf nodes like "Collie" and "Dalmatian" share the "patches" attribute, that attribute is pushed up to their parent node, "Dog."
- Learning at Multiple Abstraction Levels: Instead of a standard binary classifier, HAT uses a child-vs-parent learning scheme. To learn the "paws" attribute for a "Bear," the model trains a classifier to distinguish bear paws from the paws of its parent class ("Carnivore"). This forces the model to capture the specific visual nuances of bear paws rather than generic paws.
- Hierarchical Transfer: When evaluating a novel class, the model combines attribute classifiers from all of the class's ancestor nodes. Lower levels provide high-precision (specific) traits, while higher levels provide high-recall (general) traits, resulting in a more robust prediction.
Experimental Setup and Datasets
The model was evaluated on three standard ZSL benchmarks of varying granularity:
- aPascal/aYahoo (aPaY): 32 broad categories with 64 attributes.
- Animals with Attributes (AwA): 50 animal classes with 85 attributes.
- CUB-200-2011 Birds (CUB): A fine-grained dataset of 200 bird species with 312 attributes.
To modernize the approach, the authors also replaced traditional "shallow" computer vision features (like SIFT) with deep Convolutional Neural Network (CNN) features, such as GoogLeNet, which significantly improved baseline attribute detection.
Key Insights and Results
- Deep Features and Attributes: Upgrading to deep CNN features boosted attribute prediction accuracy (AUC) by 7% to 15% across all datasets, as higher layers of CNNs naturally capture semantic object parts.
- Hierarchy as a Precision Tool: Combining classifiers from different hierarchy levels works because they have complementary strengths. Leaf-node classifiers offer high precision, while root-node classifiers offer high recall. Averaging these predictions smooths out noise.
- State-of-the-Art Performance: Using GoogLeNet features, HAT achieved multi-class accuracy of 45.4% on aPaY, 74.9% on AwA, and 51.8% on CUB. Notably, on the fine-grained CUB dataset, HAT improved accuracy by a massive 31% over the standard Direct Attribute Prediction (DAP) baseline.
- Scalability: Unlike global models that plateau, HAT's performance increases as the source dataset grows. A denser hierarchy allows the model to learn even finer differences between sub-groups.
Conclusion and Future Directions
The HAT model demonstrates that leveraging the inherent structure of categories is vital for effective knowledge transfer in zero-shot learning. While the current model weights all hierarchy levels equally, future work may involve implementing adaptive weights to dynamically decide which abstraction level is most reliable for a specific novel class. Additionally, pulling attribute knowledge from "sibling" nodes—classes on the same branch—could provide even richer discriminative cues for classification.
Summary updated