Abstract
Zero-shot learning (ZSL) for image classification focuses on recognizing novel categories that have no labeled data available for training. The learning is generally carried out with the help of mid-level semantic descriptors associated with each class. This semantic-descriptor space is generally shared by both seen and unseen categories. However, ZSL suffers from hubness, domain discrepancy and biased-ness towards seen classes. To tackle these problems, we propose a three-step approach to zero-shot learning. Firstly, a mapping is learned from the semantic-descriptor space to the image-feature space. This mapping learns to minimize both one-to-one and pairwise distances between semantic embeddings and the image features of the corresponding classes. Secondly, we propose test-time domain adaptation to adapt the semantic embedding of the unseen classes to the test data. This is achieved by finding correspondences between the semantic descriptors and the image features. Thirdly, we propose scaled calibration on the classification scores of the seen classes. This is necessary because the ZSL model is biased towards seen classes as the unseen classes are not used in the training. Finally, to validate the proposed three-step approach, we performed experiments on four benchmark datasets where the proposed method outperformed previous results. We also studied and analyzed the performance of each component of our proposed ZSL framework.
Introduction
In traditional image recognition, models require thousands of labeled examples for every category. Zero-shot learning (ZSL) breaks this requirement by using "semantic descriptors"—such as a list of attributes like "has stripes" or "is a mammal"—to bridge the gap between seen and unseen classes. Despite its potential, ZSL faces three major hurdles:
- Hubness: In high-dimensional spaces, a few points (hubs) tend to become the "nearest neighbor" for almost every query, leading to incorrect predictions.
- Domain Discrepancy: A model trained only on "seen" classes (e.g., horses) often fails to project "unseen" classes (e.g., zebras) into the correct area of the feature space.
- Bias: In "Generalized" ZSL (GZSL), where the test set contains both seen and unseen classes, models are naturally biased toward predicting the classes they encountered during training.
Proposed Methodology
The authors introduce a robust three-step pipeline designed to mitigate these issues through relational matching, test-time adaptation, and score calibration.
1. Relational Matching
Instead of simply mapping a single attribute vector to a single image feature, the model minimizes a "structural alignment loss." This ensures that the relationships between classes (e.g., the similarity between a dog and a wolf) are preserved when transitioning from the semantic space to the visual feature space. By forcing the model to respect this relational matrix, the hubness problem is significantly reduced.
2. Test-Time Domain Adaptation
Once the model encounters unlabeled test data, it performs a rapid adaptation. Using a correspondence matrix and Group-Lasso regularization, the model "nudges" the predicted unseen class embeddings to better align with the actual distribution of the test features. This step is often the "heavy lifter" in terms of accuracy gains.
3. Scaled Calibration
To address the inherent bias toward seen classes in GZSL, the authors apply a simple scaling factor ($\gamma$) to the classification scores of seen classes. This "levels the playing field," allowing unseen classes a fair chance to be selected without requiring the entire network to be retrained.
Experimental Results
The model, referred to as OURS-RAC, was evaluated on four standard ZSL benchmarks: AwA2 (Animals), aPY (Objects), CUB (Birds), and SUN (Scenes). The results showed dramatic improvements in the Harmonic Mean (H), which balances accuracy on both seen and unseen classes.
| Dataset | Previous Best (H) | OURS-RAC (H) |
|---|---|---|
| AwA2 | 45.1% | 65.9% |
| aPY | 21.4% | 44.4% |
| SUN | 27.6% | 43.7% |
| CUB | 34.4% | 44.6% |
Qualitative analysis via t-SNE visualizations demonstrates the effectiveness of the domain adaptation step. The visualization clearly shows how the adaptation pulls "unseen" descriptors away from "seen" clusters and places them directly onto the correct unlabeled test data clusters.
Conclusion and Future Work
While the three-step approach provides a massive performance leap, challenges remain. Relational matching provided less benefit for fine-grained datasets like CUB, where bird species are highly similar and structural relations are very tight. Additionally, the current domain adaptation requires access to a batch of unlabeled test data, which may not be feasible for real-time streaming applications. Future research will focus on more sophisticated fine-grained feature extraction and online adaptation methods.
Summary updated