Abstract
Compositional Zero-Shot Learning (CZSL) aims to recognize unseen combinations of known primitives. However, real-world scenarios often involve the incremental arrival of new attributes and objects, leading to the challenge of Continual Compositional Zero-Shot Learning (CCZSL). This paper introduces a framework to handle CCZSL by addressing two primary hurdles: catastrophic forgetting and the contextual nature of primitives. We propose "Super-Primitives" to capture shared contextual traits across different categories and a "Dual Knowledge Distillation" (DKD) mechanism to balance the retention of old knowledge with the acquisition of new concepts. Evaluations on UT-Zappos and C-GQA benchmarks demonstrate that our approach significantly outperforms existing methods, establishing a robust baseline for dynamic, open-world compositional learning.
Introduction
Standard Compositional Zero-Shot Learning (CZSL) focuses on recognizing new combinations of known primitives—for example, if a model knows "red" and "car," it should recognize a "red car" even if it has never seen one. However, real-world applications are dynamic; new primitives (like the attribute "electric" or the object "drone") emerge over time. This leads to the problem of Continual Compositional Zero-Shot Learning (CCZSL), where a model must learn new attributes and objects incrementally without forgetting previously learned ones.
Existing models struggle with two main issues in a continual learning setting:
- Catastrophic Forgetting: When a model learns new data, it often "overwrites" the weights associated with old data, causing performance on previous tasks to plummet.
- Contextuality: Primitives look different depending on what they are paired with. "Old" looks different on a "dog" (gray fur) than on a "car" (rust). As new primitives are added, these contextual variations become harder to track.
Methodology
The proposed architecture is designed to be both flexible and "memorable," addressing contextuality and forgetting simultaneously.
Capturing Contextuality via Super-Primitives
Instead of learning a single static embedding for an attribute or object, the model learns Super-Attributes and Super-Objects. These are higher-level clusters (e.g., a "mammal" super-object) that capture shared contextual traits. A selection module chooses the best super-primitive to modify a base embedding, allowing the model to adapt the look of "old" based on whether the object is a car or a dog.
Dual Knowledge Distillation (DKD)
To fight forgetting, the authors use a two-way distillation process. While standard distillation helps the model remember old categories, Dual KD also constrains the model's learning on new categories. It uses the "logic" of the previous version of the model to ensure the new version doesn't become overly biased toward the most recent data session.
Experimental Evaluation
The researchers established a new evaluation protocol using two widely recognized CZSL benchmarks, adapted for incremental sessions:
- UT-Zappos: A fine-grained dataset of shoes, split into 3 incremental sessions.
- C-GQA: A large-scale dataset of natural images with hundreds of attributes and objects, split into 6 incremental sessions to simulate a more complex, long-term learning process.
Key Results
The proposed CCZSL model consistently outperformed state-of-the-art CZSL methods adapted for the continual setting:
- UT-Zappos: Achieved an average Area Under the Curve (AUC) of 30.46%, a significant 4.19% improvement over the previous best method (AoP).
- C-GQA: On this much larger dataset, the model achieved a 3.29% average AUC, outperforming the second-best method (CANet) by 1.12%.
Discussion and Conclusion
A key insight from this work is the "Consensus" of Context. While context changes appearances, there is often a "consensus" among similar items. By grouping primitives into "Super-Primitives," the model can generalize contextual rules to entirely new items. Furthermore, the paper finds that focusing on learning robust primitives is more effective for continual learning than focusing on final compositions.
While the model handles attribute-object pairs well, real-world scenes often involve multiple objects and complex relations, which this framework does not yet address. Future work could involve making the model automatically determine the number of super-primitives needed as it encounters more diverse data. Additionally, reducing the computational overhead of continual distillation remains an open challenge for scaling to thousands of primitives.
Summary updated