Learning Clusterable Visual Features for Zero-Shot Recognition

Jingyi Xu, Zhixin Shu, Dimitris SamarasarXiv 2020 · arXiv

Abstract

Zero-Shot Learning (ZSL) faces a significant challenge known as "distribution shift," where the messy visual features of real-world data conflict with the idealized synthetic features generated for unseen classes. This paper proposes a novel framework that projects real visual features into a highly clusterable embedding space. By ensuring that real features mimic the well-separated nature of generated features, the model bridges the gap between seen and unseen distributions. The approach achieves state-of-the-art performance across several ZSL and Few-Shot Learning (FSL) benchmarks, demonstrating substantial gains in Generalized Zero-Shot Learning (GZSL) accuracy.

Introduction

Zero-Shot Learning (ZSL) aims to recognize objects from categories not encountered during training by leveraging semantic descriptions or attributes. Modern ZSL techniques often employ Conditional Variational Autoencoders (CVAEs) to synthesize artificial features for these unseen classes. However, a fundamental mismatch exists: real features extracted from backbones like ResNet are often noisy and lie near decision boundaries, whereas CVAE-generated features form tight, well-separated clusters. Training a classifier on "perfect" synthetic data and testing it on "messy" real data leads to poor generalization. This research addresses this by transforming real features to be more clusterable before the generative process begins.

Methodology

The proposed pipeline focuses on learning an orderly feature space through three primary components:

  1. Discriminative Embedding Space: A mapping function ($M$) projects original ResNet features into a new space supervised by a classification loss. This forces features from the same class to group together while pushing different classes apart.
  2. Gaussian Similarity Loss: Beyond standard cross-entropy, the model utilizes a Gaussian similarity loss (based on Radial Basis Functions) to explicitly optimize for "clusterability," enhancing intra-class compactness.
  3. Gaussian Noise Injection: To prevent the classifier from overfitting to perfectly clustered data, controlled Gaussian noise is injected into the projected features during CVAE training. This improves robustness by forcing the model to handle "harder" feature variants.
Proposed architecture showing the mapping from original features to a clusterable projected space and subsequent generative reconstruction.

Experimental Setup

The method was evaluated on three standard ZSL benchmarks:

  • CUB (Caltech-UCSD Birds-200-2011): 200 fine-grained bird species.
  • SUN (Scene Attribute): 717 scene categories.
  • AWA2 (Animals with Attributes 2): 50 animal classes with 85 attributes.

Additionally, the framework was tested on Few-Shot Learning (FSL) tasks using CUB and SUN to assess the versatility of the learned clusterable features.

Results and Discussion

The proposed method significantly outperformed previous state-of-the-art (SOTA) models, particularly in Generalized Zero-Shot Learning (GZSL), where both seen and unseen classes are present at test time.

Comparison of the proposed method against existing SOTA approaches across CUB, SUN, and AWA2 datasets.

Key Performance Gains

  • Unseen Class Accuracy ($U$): The model achieved a 16.6% improvement on the SUN dataset and a 12.2% improvement on AWA2 compared to previous benchmarks.
  • Conventional ZSL: Accuracy improved by 2.9% on CUB, 2.0% on SUN, and 2.8% on AWA2.
  • Few-Shot Learning: In 1-shot scenarios, the method improved the baseline by 16.4% on CUB, highlighting the benefit of clusterable features in data-scarce regimes.

Insights

The core innovation lies in aligning real and synthetic distributions. By projecting real data into a space that mimics the clusterable nature of CVAE outputs, the distribution shift is minimized. The authors also found a strong correlation between feature "clusterability" (measured via Mutual Information) and final classification accuracy. Visualizations of the feature space confirm that while original ResNet features overlap significantly, the proposed projected features form distinct, easily separable islands.

Conclusion

This paper demonstrates that prioritizing clusterability in feature learning can effectively mitigate the distribution shift in ZSL. While the current work focuses on CVAE-based architectures, the "cluster-first" philosophy holds potential for other generative frameworks like GANs. Future research could extend this approach to other discriminative tasks such as face recognition or person re-identification, where separating highly similar identities is critical.

Summary updated

Abstract

In zero-shot learning (ZSL), conditional generators have been widely used to generate additional training features. These features can then be used to train the classifiers for testing data. However, some testing data are considered "hard" as they lie close to the decision boundaries and are prone to misclassification, leading to performance degradation for ZSL. In this paper, we propose to learn clusterable features for ZSL problems. Using a Conditional Variational Autoencoder (CVAE) as the feature generator, we project the original features to a new feature space supervised by an auxiliary classification loss. To further increase clusterability, we fine-tune the features using Gaussian similarity loss. The clusterable visual features are not only more suitable for CVAE reconstruction but are also more separable which improves classification accuracy. Moreover, we introduce Gaussian noise to enlarge the intra-class variance of the generated features, which helps to improve the classifier's robustness. Our experiments on SUN,CUB, and AWA2 datasets show consistent improvement over previous state-of-the-art ZSL results by a large margin. In addition to its effectiveness on zero-shot classification, experiments show that our method to increase feature clusterability benefits few-shot learning algorithms as well.
BibTeX
@misc{xu2020learning,
  author = {Jingyi Xu and Zhixin Shu and Dimitris Samaras},
  title = {{Learning Clusterable Visual Features for Zero-Shot Recognition}},
  year = {2020},
  eprint = {2010.03245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.03245},
}
Figure