Meta-Learned Attribute Self-Interaction Network for Continual and Generalized Zero-Shot Learning

Vinay Kumar Verma, Nikhil Mehta, Kevin J. Liang, Aakansha Mishra, Lawrence Carin

Abstract

Zero-shot learning (ZSL) is a promising approach to generalizing a model to categories unseen during training by leveraging class attributes, but challenges remain. Recently, methods using generative models to combat bias towards classes seen during training have pushed state of the art, but these generative models can be slow or computationally expensive to train. Also, these generative models assume that the attribute vector of each unseen class is available a priori at training, which is not always practical. Additionally, while many previous ZSL methods assume a one-time adaptation to unseen classes, in reality, the world is always changing, necessitating a constant adjustment of deployed models. Models unprepared to handle a sequential stream of data are likely to experience catastrophic forgetting. We propose a Meta-learned Attribute self-Interaction Network (MAIN) for continual ZSL. By pairing attribute self-interaction trained using meta-learning with inverse regularization of the attribute encoder, we are able to outperform state-of-the-art results without leveraging the unseen class attributes while also being able to train our models substantially faster (>100x) than expensive generative-based approaches. We demonstrate this with experiments on five standard ZSL datasets (CUB, aPY, AWA1, AWA2, and SUN) in the generalized zero-shot learning and continual (fixed/dynamic) zero-shot learning settings. Extensive ablations and analyses demonstrate the efficacy of various components proposed.

Introduction

Zero-shot learning (ZSL) represents a significant milestone in artificial intelligence, enabling models to recognize novel categories by leveraging semantic descriptions rather than visual examples. However, current state-of-the-art approaches often rely on complex generative models (like GANs or VAEs) that suffer from two critical flaws: they require prior knowledge of unseen class attributes during training, and they are computationally expensive. Furthermore, these models are typically designed for static environments, making them susceptible to "catastrophic forgetting" when faced with a continuous stream of new data.

To address these limitations, we introduce the Meta-learned Attribute self-Interaction Network (MAIN). MAIN is a non-generative framework designed for both Generalized ZSL (GZSL) and Continual ZSL (CZSL), offering a training speedup of up to 300x while achieving superior performance without requiring future knowledge of unseen classes.

The MAIN Framework

MAIN moves away from simple linear mappings, instead focusing on capturing high-order relationships within class descriptions through a novel architecture.

Architecture of the Meta-learned Attribute self-Interaction Network (MAIN), highlighting the self-interaction module and the probabilistic inverse path.

Core Innovations

  1. Self-Interaction Module (SIA): Rather than treating attributes independently, the SIA uses a self-gating mechanism. This allows the model to learn complex interactions (e.g., how "wings" and "small size" together define a specific bird category), creating a more robust internal representation.
  2. Inverse Regularization (IR): To prevent overfitting to "seen" classes, MAIN employs a cyclic loss that maps visual features back to semantic attributes. This process is mathematically equivalent to maximizing entropy in the visual embedding space, which significantly improves generalization to unseen categories.
  3. Meta-Learning with Reptile: By utilizing the Reptile algorithm, MAIN trains its attribute encoder to be highly adaptable. This prepares the model to generalize to novel classes from minimal information without requiring extensive fine-tuning.
  4. Reservoir Sampling: To mitigate catastrophic forgetting in continual learning settings, a small reservoir of samples from previous tasks is replayed during new training sessions, anchoring the model's memory.

Experimental Results

We evaluated MAIN across five standard benchmarks: CUB, AWA1, AWA2, SUN, and aPY. The experiments covered standard GZSL as well as the more challenging Fixed and Dynamic Continual GZSL settings.

Performance and Efficiency

MAIN demonstrates remarkable efficiency, reducing training time from hours to seconds. For instance, on the AWA2 dataset, MAIN trains in approximately 45 seconds compared to 3 hours for generative baselines.

In the Dynamic Continual GZSL setting—the most realistic and difficult scenario—MAIN outperformed existing methods by substantial margins. On the CUB dataset, it achieved a 13.41% absolute gain in harmonic mean over previous state-of-the-art approaches.

Performance comparison of MAIN against baseline methods across sequential tasks, demonstrating superior stability and higher harmonic mean.

Key Insights

  • Entropy and Generalization: The theoretical link between Inverse Regularization and entropy maximization explains why MAIN avoids the common bias toward seen classes without needing synthetic data generation.
  • Non-linear Interaction: Ablation studies confirmed that non-linear self-gating (using ReLU and Sigmoid) is more effective than standard polynomial kernels for capturing the nuances of class attributes.
  • Generative Mismatch: Interestingly, adding synthetic samples from generative models to MAIN actually decreased performance, suggesting a fundamental mismatch between real visual features and current synthetic outputs.

Conclusion

MAIN provides a fast, efficient, and robust solution for real-world zero-shot learning. By combining meta-learning with attribute self-interaction and mathematically grounded regularization, it bridges the gap between high-performance research models and the practical needs of evolving, data-rich environments.

Summary updated

Abstract

Zero-shot learning (ZSL) is a promising approach to generalizing a model to categories unseen during training by leveraging class attributes, but challenges remain. Recently, methods using generative models to combat bias towards classes seen during training have pushed state of the art, but these generative models can be slow or computationally expensive to train. Also, these generative models assume that the attribute vector of each unseen class is available a priori at training, which is not always practical. Additionally, while many previous ZSL methods assume a one-time adaptation to unseen classes, in reality, the world is always changing, necessitating a constant adjustment of deployed models. Models unprepared to handle a sequential stream of data are likely to experience catastrophic forgetting. We propose a Meta-learned Attribute self-Interaction Network (MAIN) for continual ZSL. By pairing attribute self-interaction trained using meta-learning with inverse regularization of the attribute encoder, we are able to outperform state-of-the-art results without leveraging the unseen class attributes while also being able to train our models substantially faster (>100x) than expensive generative-based approaches. We demonstrate this with experiments on five standard ZSL datasets (CUB, aPY, AWA1, AWA2, and SUN) in the generalized zero-shot learning and continual (fixed/dynamic) zero-shot learning settings. Extensive ablations and analyses demonstrate the efficacy of various components proposed.
BibTeX
@inproceedings{verma2024meta,
  author = {Vinay Kumar Verma and Nikhil Mehta and Kevin J. Liang and Aakansha Mishra and Lawrence Carin},
  title = {{Meta-Learned Attribute Self-Interaction Network for Continual and Generalized Zero-Shot Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=6qDjVEpN3I},
}
Figure