CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning

Emanuele Frascaroli, Aniello Panariello, Pietro Buzzega, Lorenzo Bonicelli, Angelo Porrello, Simone Calderara

Abstract

Vision-Language Models (VLMs) like CLIP demonstrate impressive zero-shot capabilities but are susceptible to catastrophic forgetting when fine-tuned on sequential tasks. This paper introduces CGIL (Continual Generative training for Incremental prompt-Learning), a novel framework that mitigates forgetting by replaying synthetic visual features. By utilizing lightweight class-specific Variational Autoencoders (VAEs) to model the latent embedding space, CGIL maintains performance on previous tasks while adapting to new domains. Experimental results across diverse benchmarks show that CGIL significantly outperforms existing state-of-the-art methods, achieving an average accuracy improvement of +11% and enhancing zero-shot transfer to future tasks.

Introduction

Continual Learning (CL) remains a fundamental challenge in artificial intelligence, particularly for large-scale pre-trained models. When models are updated with new data, they often overwrite the parameters necessary for previous tasks—a phenomenon known as catastrophic forgetting. While Vision-Language Models (VLMs) like CLIP provide a robust foundation for general visual recognition, adapting them to specialized domains (e.g., medical imaging or remote sensing) typically degrades their original zero-shot utility. Existing prompt-tuning methods attempt to address this but often struggle to balance plasticity for new knowledge with stability for old knowledge.

Methodology: The CGIL Framework

The authors propose CGIL, which combines prompt learning with a generative latent replay mechanism. Unlike traditional replay methods that store raw images—raising privacy and storage concerns—CGIL operates entirely within CLIP’s latent embedding space.

Two-Phase Training

  1. Generative Modeling Phase: For every new category encountered, the system trains a class-specific VAE. This VAE (approximately 0.5M parameters) learns the distribution of that class within the frozen CLIP visual embedding space.
  2. Prompt Alignment Phase: To learn the current task, the model "replays" synthetic visual embeddings from the VAE decoders of all previously seen classes. It then optimizes class-specific textual prompts to align with these synthetic embeddings.
The CGIL workflow: training class-specific VAEs (left) and using them to align textual prompts via synthetic replay (right).

During this process, the CLIP encoders remain frozen. The model learns a hybrid prompt consisting of a class-specific token ($V_c$) and a "hyper-token" ($V_c^H$) generated by a small MLP. This architecture ensures the model retains the broad knowledge of the original CLIP while specializing for new tasks.

Experimental Results

CGIL was evaluated on five diverse benchmarks: Split ImageNet-R, Split Cars-196, Split CUB-200, Split EuroSAT (satellite imagery), and Split ISIC (medical imaging).

State-of-the-Art Performance

CGIL consistently outperformed existing methods like SLCA and MoE Adapters. Notably, in specialized domains such as EuroSAT and ISIC, where the data distribution differs significantly from CLIP’s pre-training data, CGIL showed substantial gains. On average, CGIL achieved a final accuracy of 86.20%, an 11% absolute improvement over the previous best-performing baseline.

Comparison of Final Average Accuracy across various benchmarks. CGIL consistently takes the lead, especially in specialized domains like EuroSAT and ISIC.

CI-Transfer: Preserving Zero-Shot Capabilities

The researchers also introduced the CI-Transfer metric to evaluate how incremental learning affects a model's ability to recognize future, unseen classes. CGIL demonstrated a unique ability to leverage learned knowledge to improve zero-shot performance over time, outperforming other VLM-based approaches by a wide margin. This suggests that the generative replay mechanism helps the model maintain a more coherent and generalized embedding space.

Conclusion and Future Work

CGIL establishes a strong baseline for incremental learning in VLMs by leveraging the efficiency of latent space generative replay. While the method is highly effective, the requirement to store one VAE decoder per class may present scaling challenges for datasets with thousands of categories. Future research will focus on compressing the decoder storage and further optimizing the computational overhead of the prompt alignment phase.

Summary updated

Abstract

Abstract
BibTeX
@inproceedings{frascaroli2024clip,
  author = {Emanuele Frascaroli and Aniello Panariello and Pietro Buzzega and Lorenzo Bonicelli and Angelo Porrello and Simone Calderara},
  title = {{CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning}},
  booktitle = {BMVC},
  year = {2024},
  url = {https://openreview.net/forum?id=0bWktn8Gxz},
}
Figure