Zero-shot Image Recognition Using Relational Matching, Adaptation and Calibration

Debasmit Das, C. S. George Lee

TL;DR

This paper introduces a robust three-step framework to improve Zero-Shot Learning (ZSL)—the ability of a model to recognize objects it has never seen during training. By combining relational matching (preserving class structures), test-time domain adaptation, and score calibration, the authors achieved a massive performance leap (up to 23%) over previous state-of-the-art methods on standard benchmarks.

What Problem Does This Paper Address?

In traditional image recognition, you need thousands of labeled examples for every category. Zero-shot learning breaks this requirement by using "semantic descriptors" (like a list of attributes: "has stripes," "is a mammal") to bridge the gap between seen and unseen classes.

However, ZSL faces three major hurdles:

  1. Hubness: In high-dimensional spaces, a few points (hubs) tend to become the "nearest neighbor" for almost every query, leading to wrong predictions.
  2. Domain Discrepancy: A model trained only on "seen" classes (e.g., horses) often fails to project "unseen" classes (e.g., zebras) into the correct area of the feature space.
  3. Bias: In "Generalized" ZSL (where the test set contains both seen and unseen classes), models are naturally biased toward predicting the classes they saw during training.

Methods

The authors propose a three-step pipeline to tackle these issues:

  1. Relational Matching: Instead of just mapping a single attribute vector to a single image feature, they minimize a "structural alignment loss." This ensures that the relationships between classes (e.g., how similar a dog is to a wolf) are preserved when moving from the semantic space to the visual feature space.
  2. Test-Time Domain Adaptation: Once the model sees the unlabeled test data, it performs a quick adaptation. It uses a correspondence matrix and Group-Lasso regularization to "nudge" the predicted unseen class embeddings so they better align with the actual distribution of the test features.
  3. Scaled Calibration: To fix the bias toward seen classes, the authors apply a simple scaling factor ($\gamma$) to the classification scores of seen classes. This effectively "levels the playing field," allowing unseen classes a fair chance to be selected as the winner.

Datasets

The model was evaluated on four standard ZSL benchmarks:

  • AwA2 (Animals with Attributes 2): 50 animal classes, 37,322 images.
  • aPY (Attribute Pascal and Yahoo): 32 classes including vehicles and animals.
  • CUB (Caltech-UCSD Birds-200-2011): A fine-grained dataset of 200 bird species.
  • SUN (Scene UNderstanding): 717 scene categories.

Key Insights and Ideas

  • Structure Matters: Mapping individual points isn't enough. By forcing the model to respect the "relational matrix" (the web of similarities between all classes), the authors significantly reduced the hubness problem.
  • Adaptation is the "Heavy Lifter": The results show that domain adaptation provided the biggest boost in accuracy. It transforms the unseen class embeddings from being "near" seen classes to being at the center of their own actual feature clusters.
  • Calibration is Essential for GZSL: Without calibration, even a great model will often misclassify a zebra as a horse simply because it "knows" horses better. A simple multiplier can balance this out without retraining the whole network.

Key Results

The proposed method, OURS-RAC, showed dramatic improvements in the Harmonic Mean (H)—a metric that balances accuracy on both seen and unseen classes:

  • AwA2: Jumped from 45.1% (previous best) to 65.9%.
  • aPY: Jumped from 21.4% to 44.4%.
  • SUN: Jumped from 27.6% to 43.7%.
  • CUB: Improved from 34.4% to 44.6%.

Qualitatively, t-SNE visualizations (Fig. 3) clearly show how the domain adaptation step pulls the "unseen" descriptors away from the "seen" clusters and places them directly onto the correct unlabeled test data clusters.

Limitations and Future Work

  • Fine-Grained Challenges: The authors noted that relational matching (Step 1) provided less benefit for the CUB (birds) dataset. Because bird species are so similar, their structural relations are very "tight," making it harder for global relational matching to provide a clear signal.
  • Transductive Requirement: The domain adaptation step requires access to a batch of unlabeled test data at once. This might be difficult in "streaming" applications where images arrive one by one and must be classified instantly.
  • Future Directions: The authors suggest exploring more sophisticated ways to extract fine-grained features to help with datasets like CUB.

Summary updated

Abstract

Zero-shot learning (ZSL) for image classification focuses on recognizing novel categories that have no labeled data available for training. The learning is generally carried out with the help of mid-level semantic descriptors associated with each class. This semantic-descriptor space is generally shared by both seen and unseen categories. However, ZSL suffers from hubness, domain discrepancy and biased-ness towards seen classes. To tackle these problems, we propose a three-step approach to zero-shot learning. Firstly, a mapping is learned from the semantic-descriptor space to the image-feature space. This mapping learns to minimize both one-to-one and pairwise distances between semantic embeddings and the image features of the corresponding classes. Secondly, we propose test-time domain adaptation to adapt the semantic embedding of the unseen classes to the test data. This is achieved by finding correspondences between the semantic descriptors and the image features. Thirdly, we propose scaled calibration on the classification scores of the seen classes. This is necessary because the ZSL model is biased towards seen classes as the unseen classes are not used in the training. Finally, to validate the proposed three-step approach, we performed experiments on four benchmark datasets where the proposed method outperformed previous results. We also studied and analyzed the performance of each component of our proposed ZSL framework.
BibTeX
@inproceedings{das2019zero,
  author = {Debasmit Das and C. S. George Lee},
  title = {{Zero-shot Image Recognition Using Relational Matching, Adaptation and Calibration}},
  booktitle = {IJCNN},
  year = {2019},
  url = {https://openreview.net/forum?id=SW4UWTnXGQ5},
}

Loading related content...

Figure