Continual Compositional Zero-Shot Learning

Yang Zhang, Songhe Feng, Jiazheng Yuan

Introduction

Traditional Compositional Zero-Shot Learning (CZSL) focuses on recognizing new combinations of known primitives—for example, recognizing a "red bike" after seeing "red cars" and "blue bikes." However, real-world scenarios are rarely static. New concepts emerge constantly, requiring models to learn new attributes (like "metallic") or objects (like "drone") incrementally. This paper introduces Continual Compositional Zero-Shot Learning (CCZSL), a benchmark and framework designed for models that must evolve without retraining from scratch.

The Challenge of Incremental Learning

The authors identify two major hurdles in the CCZSL setting:

  1. Contextuality: Primitives look different depending on their pair. "Old" has a different visual signature on a dog than on a car. In a continual setting, the model must adapt to these shifting signatures as new pairs are introduced.
  2. Catastrophic Forgetting: As models learn new primitives in later sessions, they tend to overwrite the weights for older ones, leading to a sharp performance drop on earlier tasks.
Figure 1: The CCZSL task setup. Primitives are introduced incrementally across sessions, expanding the space of possible seen and unseen compositions over time.

Proposed Methodology

The authors propose a framework consisting of two primary innovations to handle context and memory.

1. Super-Primitives for Contextuality

Instead of static embeddings, the model learns a set of Super-Primitives that act as contextual templates. A Selection Module determines which Super-Primitive best fits the current image, which then modifies the invariant "base" primitive embedding to match the specific visual context (e.g., applying a "mammal" context to the attribute "wet").

2. Dual Knowledge Distillation (DKD)

To prevent forgetting, the framework uses Dual KD. While standard distillation focuses on keeping old class outputs consistent, DKD also applies distillation to new categories. This prevents the model from becoming overly biased toward the most recently learned data, ensuring a balanced prediction space across all sessions.

Figure 3: The CCZSL architecture. (a) Shows the Super-Primitive selection and transformation modules for capturing context. (b) Illustrates the continual learning pipeline using Knowledge Distillation (L_KD) to maintain stability between sessions.

Experimental Results

The researchers adapted two standard benchmarks, UT-Zappos (shoes) and C-GQA (natural images), into incremental versions with multiple learning sessions.

  • UT-Zappos: CCZSL achieved an average AUC of 30.46%, a 4.19% improvement over the best-performing baseline (AoP).
  • C-GQA: On this complex dataset, CCZSL reached an average AUC of 3.29%, outperforming the next best model (CANet) by 1.12%.
  • Stability: While baseline models often saw performance plummet in later sessions, CCZSL maintained significantly higher performance, proving its resistance to catastrophic forgetting.

Conclusion

By leveraging Super-Primitives to handle visual context and Dual Knowledge Distillation to preserve memory, the CCZSL framework successfully addresses the complexities of evolving visual vocabularies. This work moves zero-shot learning closer to real-world applications where AI must learn and adapt continuously.

Summary updated

BibTeX
@inproceedings{zhang2024continual,
  author = {Yang Zhang and Songhe Feng and Jiazheng Yuan},
  title = {{Continual Compositional Zero-Shot Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure