Abstract
Zero-shot learning (ZSL) is a promising approach to generalizing a model to categories unseen during training by leveraging class attributes, but challenges remain. Recently, methods using generative models to combat bias towards classes seen during training have pushed state of the art, but these generative models can be slow or computationally expensive to train. Also, these generative models assume that the attribute vector of each unseen class is available a priori at training, which is not always practical. Additionally, while many previous ZSL methods assume a one-time adaptation to unseen classes, in reality, the world is always changing, necessitating a constant adjustment of deployed models. Models unprepared to handle a sequential stream of data are likely to experience catastrophic forgetting. We propose a Meta-learned Attribute self-Interaction Network (MAIN) for continual ZSL. By pairing attribute self-interaction trained using meta-learning with inverse regularization of the attribute encoder, we are able to outperform state-of-the-art results without leveraging the unseen class attributes while also being able to train our models substantially faster (> 100×) than expensive generative-based approaches. We demonstrate this with experiments on five standard ZSL datasets (CUB, aPY, AWA1, AWA2, and SUN) in the generalized zero-shot learning and continual (fixed/dynamic) zero-shot learning settings. Extensive ablations and analyses demonstrate the efficacy of various components proposed.
Introduction
Traditional Zero-Shot Learning (ZSL) aims to recognize objects from categories never seen during training by leveraging semantic "attributes" (e.g., "has wings," "is yellow"). Despite its promise, the field faces three major hurdles. First, the generative bottleneck: current top-performing models rely on Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) to synthesize data for unseen classes, which is computationally expensive. Second, unrealistic assumptions: most generative models require attribute vectors of unseen classes during training, a requirement often impossible in real-world scenarios. Finally, catastrophic forgetting: in continual settings where new classes arrive sequentially, models tend to forget previously learned knowledge.
To address these, we present MAIN (Meta-learned Attribute self-Interaction Network), a non-generative framework that is up to 300x faster to train than generative counterparts and does not require prior knowledge of unseen class attributes.
The MAIN Framework
The MAIN architecture is built upon three core pillars designed to enhance generalization and efficiency:
1. Self-Interaction for Attributes (SIA)
Attributes are rarely independent. MAIN uses a specialized module to model complex relationships within the attribute vector. We employ Self-Gating, which uses a sigmoid-based gate to weight the importance of specific attribute dimensions, and Polynomial Kernels to approximate high-order interactions. This provides a much richer representation for the visual-semantic bridge than simple linear mappings.
2. Inverse Regularization (IR)
To prevent overfitting to "seen" classes, we introduce a cyclic consistency loss. This forces the model to map a visual embedding back to its original semantic attribute space ($t_\psi$). Theoretically, this maximizes entropy in the visual embedding space, which significantly aids generalization to unseen categories.
3. Meta-Learning with Reptile
We treat ZSL as a "learning-to-learn" problem using the Reptile algorithm. This first-order meta-learning approach allows the attribute encoder to find a parameter initialization that generalizes naturally to novel classes without requiring explicit fine-tuning on them.
Continual Zero-Shot Learning
In real-world applications, data arrives in a stream. We evaluate MAIN in both Fixed Continual GZSL (fixed sequence of classes) and Dynamic Continual GZSL (where both seen and unseen classes increase over time). To mitigate catastrophic forgetting, MAIN maintains a tiny, constant-sized reservoir buffer of samples from previous tasks to "replay" during training.
Key Results
MAIN was evaluated on five standard benchmarks: CUB, AWA1, AWA2, SUN, and aPY.
- Efficiency: Training MAIN takes roughly 45 seconds, compared to approximately 3 hours for generative competitors—a speedup of over 100x.
- Performance: In the Dynamic Continual GZSL setting on the CUB dataset, MAIN achieved a harmonic mean (mH) of 67.20%, representing a 13.41% absolute improvement over the previous best baseline (NM-ZSL).
- Robustness: Even without seeing unseen attributes during training, MAIN matches or exceeds the performance of expensive generative models on SUN, AWA1, and AWA2.
Conclusion
MAIN demonstrates that expensive generative models are not a prerequisite for state-of-the-art Zero-Shot Learning. By combining attribute self-interaction, meta-learning, and inverse regularization, we achieve a model that is significantly faster, more realistic in its training assumptions, and highly robust to the challenges of continual learning.
Summary updated