Continual Compositional Zero-Shot Learning

Yang Zhang, Songhe Feng, Jiazheng Yuan, Xiaobin Hu, Boyuan Jiang, Chengming Xu, Taisong Jin, Chengjie Wang, Yanwei Fu

Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen combinations of known primitives. However, real-world scenarios often involve the incremental arrival of new attributes and objects, leading to the challenge of Continual Compositional Zero-Shot Learning (CCZSL). This paper introduces a framework to handle CCZSL by addressing two primary hurdles: catastrophic forgetting and the contextual nature of primitives. We propose "Super-Primitives" to capture shared contextual traits across different categories and a "Dual Knowledge Distillation" (DKD) mechanism to balance the retention of old knowledge with the acquisition of new concepts. Evaluations on UT-Zappos and C-GQA benchmarks demonstrate that our approach significantly outperforms existing methods, establishing a robust baseline for dynamic, open-world compositional learning.

Introduction

Standard Compositional Zero-Shot Learning (CZSL) focuses on recognizing new combinations of known primitives—for example, if a model knows "red" and "car," it should recognize a "red car" even if it has never seen one. However, real-world applications are dynamic; new primitives (like the attribute "electric" or the object "drone") emerge over time. This leads to the problem of Continual Compositional Zero-Shot Learning (CCZSL), where a model must learn new attributes and objects incrementally without forgetting previously learned ones.

Illustration of the CCZSL task across multiple sessions, where new objects and attributes are introduced over time.

Existing models struggle with two main issues in a continual learning setting:

  1. Catastrophic Forgetting: When a model learns new data, it often "overwrites" the weights associated with old data, causing performance on previous tasks to plummet.
  2. Contextuality: Primitives look different depending on what they are paired with. "Old" looks different on a "dog" (gray fur) than on a "car" (rust). As new primitives are added, these contextual variations become harder to track.

Methodology

The proposed architecture is designed to be both flexible and "memorable," addressing contextuality and forgetting simultaneously.

Capturing Contextuality via Super-Primitives

Instead of learning a single static embedding for an attribute or object, the model learns Super-Attributes and Super-Objects. These are higher-level clusters (e.g., a "mammal" super-object) that capture shared contextual traits. A selection module chooses the best super-primitive to modify a base embedding, allowing the model to adapt the look of "old" based on whether the object is a car or a dog.

Dual Knowledge Distillation (DKD)

To fight forgetting, the authors use a two-way distillation process. While standard distillation helps the model remember old categories, Dual KD also constrains the model's learning on new categories. It uses the "logic" of the previous version of the model to ensure the new version doesn't become overly biased toward the most recent data session.

The proposed model architecture. Part (a) shows the feature encoding and super-primitive selection, while part (b) illustrates the continual learning process with Dual Knowledge Distillation.

Experimental Evaluation

The researchers established a new evaluation protocol using two widely recognized CZSL benchmarks, adapted for incremental sessions:

  • UT-Zappos: A fine-grained dataset of shoes, split into 3 incremental sessions.
  • C-GQA: A large-scale dataset of natural images with hundreds of attributes and objects, split into 6 incremental sessions to simulate a more complex, long-term learning process.

Key Results

The proposed CCZSL model consistently outperformed state-of-the-art CZSL methods adapted for the continual setting:

  • UT-Zappos: Achieved an average Area Under the Curve (AUC) of 30.46%, a significant 4.19% improvement over the previous best method (AoP).
  • C-GQA: On this much larger dataset, the model achieved a 3.29% average AUC, outperforming the second-best method (CANet) by 1.12%.

Discussion and Conclusion

A key insight from this work is the "Consensus" of Context. While context changes appearances, there is often a "consensus" among similar items. By grouping primitives into "Super-Primitives," the model can generalize contextual rules to entirely new items. Furthermore, the paper finds that focusing on learning robust primitives is more effective for continual learning than focusing on final compositions.

While the model handles attribute-object pairs well, real-world scenes often involve multiple objects and complex relations, which this framework does not yet address. Future work could involve making the model automatically determine the number of super-primitives needed as it encounters more diverse data. Additionally, reducing the computational overhead of continual distillation remains an open challenge for scaling to thousands of primitives.

Summary updated

BibTeX
@inproceedings{zhang2024continual,
  author = {Yang Zhang and Songhe Feng and Jiazheng Yuan and Xiaobin Hu and Boyuan Jiang and Chengming Xu and Taisong Jin and Chengjie Wang and Yanwei Fu},
  title = {{Continual Compositional Zero-Shot Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure