Unseen Classes at a Later Time? No Problem

Hari Chandana Kuchibhotla, Sumitra S. Malagi, Shivam Chandhok, Vineeth N. Balasubramanian

Abstract

Recent progress towards learning from limited supervision has encouraged efforts towards designing models that can recognize novel classes at test time (generalized zero-shot learning or GZSL). GZSL approaches assume knowledge of all classes, with or without labeled data, beforehand. However, practical scenarios demand models that are adaptable and can handle dynamic addition of new seen and unseen classes on the fly (i.e continual generalized zero-shot learning or CGZSL). One solution is to sequentially retrain and reuse conventional GZSL methods, however, such an approach suffers from catastrophic forgetting leading to suboptimal generalization performance. A few recent efforts towards tackling CGZSL have been limited by difference in settings, practicality, data splits and protocols followed - inhibiting fair comparison and a clear direction forward. Motivated from these observations, in this work, we firstly consolidate the different CGZSL setting variants and propose a new Online-CGZSL setting which is more practical and flexible. Secondly, we introduce a unified feature-generative framework for CGZSL that leverages bi-directional incremental alignment to dynamically adapt to addition of new classes, with or without labeled data, that arrive over time in any of these CGZSL settings. Our comprehensive experiments and analysis on five benchmark datasets and comparison with baselines show that our approach consistently outperforms existing methods, especially on the more practical Online setting.

Introduction

Traditional Generalized Zero-Shot Learning (GZSL) enables models to recognize "unseen" classes by leveraging semantic attributes. However, standard GZSL assumes a static environment where the entire pool of classes is known from the outset. In real-world applications, data arrives in a stream, requiring models to learn new classes sequentially without forgetting previously acquired knowledge.

Previous attempts at Continual GZSL (CGZSL) have been restricted by rigid protocols. For instance, they often fail to account for the transition of an "unseen" class to a "seen" class once labeled data becomes available. This paper addresses these limitations by formalizing the Online-CGZSL setting and proposing a framework to mitigate catastrophic forgetting.

Comparison of different learning settings and the proposed CGZSL variants.

Methodology

The authors propose a unified feature-generative framework. Instead of direct image classification, the model learns to generate synthetic visual features from semantic attributes using a Generative Adversarial Network (GAN).

Unified Feature-Generative Framework

The framework consists of three main components: training, inference, and generative replay.

  • Online-CGZSL Setting: A flexible protocol where tasks arrive sequentially, allowing for "unseen-to-seen" conversion, mirroring real-world discovery and labeling.
  • Cosine Similarity-based GAN Classifier: To handle a dynamic number of classes, the model utilizes cosine similarity between visual features and attribute projections, avoiding the constraints of fixed-size linear layers.
  • Bi-directional Incremental Alignment: This mechanism uses Semantic Alignment ($L_{sal}$) to ensure visual feature distances mirror semantic attribute distances, and Nuclear Loss ($L_{nuclear}$) to align generated feature means with attribute projections.
  • Generative Replay: To prevent catastrophic forgetting, the model's generator "replays" features of previously learned classes, eliminating the need to store raw historical data.
The proposed unified feature-generative framework for CGZSL.

Experiments and Results

The model was evaluated on five major ZSL benchmarks: AWA1, AWA2, CUB, SUN, and aPY. It was tested across three settings: Static-CGZSL, Dynamic-CGZSL, and the newly proposed Online-CGZSL.

The proposed approach consistently outperformed state-of-the-art baselines:

  • Online-CGZSL Performance: On the AWA2 dataset, the model achieved a 75.44% Harmonic Mean (mH), significantly higher than the next best baseline (NM-ZSL at 61.91%).
  • Generalization Gains: Across all datasets in the Online setting, the model showed mH improvements ranging from 3.04% to 10.54%.
  • Ablation Studies: Removing bi-directional alignment losses resulted in performance drops of up to 28%, highlighting the importance of semantic-visual space alignment for continual zero-shot success.

Conclusion

This work introduces a more realistic Online-CGZSL setting and a robust feature-generative framework to address the challenges of continual learning in a zero-shot context. By leveraging bi-directional incremental alignment and generative replay, the model effectively mitigates catastrophic forgetting and achieves superior performance on benchmark datasets. Future work may explore dynamic attention mechanisms to further enhance adaptability to complex visual changes over time.

Summary updated

Abstract

Recent progress towards learning from limited supervision has encouraged efforts towards designing models that can recognize novel classes at test time (generalized zero-shot learning or GZSL). GZSL approaches assume knowledge of all classes, with or without labeled data, beforehand. However, practical scenarios demand models that are adaptable and can handle dynamic addition of new seen and unseen classes on the fly (i.e continual generalized zero-shot learning or CGZSL). One solution is to sequentially retrain and reuse conventional GZSL methods, however, such an approach suffers from catastrophic forgetting leading to suboptimal generalization performance. A few recent efforts towards tackling CGZSL have been limited by difference in settings, practicality, data splits and protocols followed - inhibiting fair comparison and a clear direction forward. Motivated from these observations, in this work, we firstly consolidate the different CGZSL setting variants and propose a new Online-CGZSL setting which is more practical and flexible. Secondly, we introduce a unified feature-generative framework for CGZSL that leverages bi-directional incremental alignment to dynamically adapt to addition of new classes, with or without labeled data, that arrive over time in any of these CGZSL settings. Our comprehensive experiments and analysis on five benchmark datasets and comparison with baselines show that our approach consistently outperforms existing methods, especially on the more practical Online setting.
BibTeX
@inproceedings{kuchibhotla2022unseen,
  author = {Hari Chandana Kuchibhotla and Sumitra S. Malagi and Shivam Chandhok and Vineeth N. Balasubramanian},
  title = {{Unseen Classes at a Later Time? No Problem}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=2m83FeOWE7},
}
Figure