Continual Learning in Cross-Modal Retrieval

Kai Wang, Luis Herranz, Joost van de Weijer

Abstract: Multimodal representations and continual learning are two areas closely related to human intelligence. The former considers the learning of shared representation spaces where information from different modalities can be compared and integrated (we focus on cross-modal retrieval between language and visual representations). The latter studies how to prevent forgetting a previously learned task when learning a new one. While humans excel in these two aspects, deep neural networks are still quite limited. In this paper, we propose a combination of both problems into a continual cross-modal retrieval setting, where we study how the catastrophic interference caused by new tasks impacts the embedding spaces and their cross-modal alignment required for effective retrieval. We propose a general framework that decouples the training, indexing and querying stages. We also identify and study different factors that may lead to forgetting, and propose tools to alleviate it. We found that the indexing stage pays an important role and that simply avoiding reindexing the database with updated embedding networks can lead to significant gains. We evaluated our methods in two image-text retrieval datasets, obtaining significant gains with respect to the fine tuning baseline.

Introduction

Standard cross-modal retrieval systems aim to map disparate data types, such as images and text, into a shared embedding space where semantically similar items reside close together. However, most research assumes a static environment. In real-world applications, an agent must learn new concepts sequentially—for instance, learning about "animals" today and "vehicles" tomorrow.

When deep neural networks learn new tasks, they typically suffer from catastrophic forgetting, where weight updates for new information degrade performance on previously learned tasks. In a cross-modal retrieval setting, this problem is compounded: the model must not only retain old features but also maintain the precise alignment between the image and text branches. If these branches drift at different rates or in different directions, the shared embedding space collapses, leading to retrieval failure.

A Framework for Continual Retrieval

The authors propose a general framework that decouples the retrieval process into three distinct stages: training, indexing, and querying. This separation allows for more granular control over how the system "remembers" old information.

General framework for continual cross-modal retrieval showing the training, indexing, and querying stages.
  1. Training: Learning feature extractors ($f_{\theta}$ for images, $g_{\omega}$ for text) for the current task using a bi-directional ranking loss.
  2. Indexing: Processing the database to store embeddings ($I_t$) for the current task.
  3. Querying: Using a query (e.g., text) to find the best match in the accumulated index ($I_1, \dots, I_t$).

Strategies to Mitigate Forgetting

To combat catastrophic forgetting, the paper evaluates several strategies beyond standard fine-tuning:

  • Weight Regularization: Adapting techniques like Elastic Weight Consolidation (EWC) and Memory Aware Synapses (MAS) to the retrieval loss to penalize changes to parameters critical for previous tasks.
  • Layer Sharing: Sharing the top layers of the image and text branches forces both modalities to "drift together," which helps maintain cross-modal alignment even as the model evolves.
  • The "No Reindexing" Strategy: Traditionally, when a model is updated, the entire database is re-processed to generate the most "accurate" embeddings. However, the authors found that keeping old embeddings fixed (no reindexing) acts as an anchor. This creates an asymmetric retrieval (new query vs. old index) that prevents old data from being distorted by new gradients.
Comparison of indexing and querying strategies, highlighting the asymmetric 'no reindexing' approach (b) versus standard reindexing.

Experimental Results and Insights

The methods were evaluated on two new benchmarks: SeViGe (Sequential Visual Genome), split into animals, vehicles, and clothes; and SeCOCO (Sequential MS-COCO), organized by super-category groupings.

Key Findings

  • Indexing is a Critical Lever: The state of the database (the index) is as vital to the system's memory as the model weights themselves. Simply avoiding reindexing led to performance boosts of 3% to 5% in Recall@10 on the SeViGe dataset.
  • Unequal Drift is the Primary Enemy: Forgetting in these systems is often caused by the image and text branches drifting in different directions. This misalignment is frequently more damaging than the loss of discriminative power within a single modality.
  • Synergistic Effects: The best performance was achieved by combining the "no reindexing" strategy with shared top layers and MAS regularization.
  • Task Knowledge: Performance improves significantly if the system knows which task a query belongs to (task-aware), as this prevents different tasks from cluttering the same regions of the embedding space.

Limitations and Future Work

While these strategies alleviate forgetting, cross-task overlap remains a challenge. Without "Cross-Task Negative Pairs" (comparing new images to old text during training), the model struggles to keep distinct tasks separated in the shared space. Future research may explore pseudo-rehearsal using generative models to synthesize examples of old tasks, maintaining separation without the need to store raw historical data. Additionally, the scalability of asymmetric "no reindexing" across a very large number of sequential tasks warrants further investigation.

Summary updated

Abstract

Multimodal representations and continual learning are two areas closely related to human intelligence. The former considers the learning of shared representation spaces where information from different modalities can be compared and integrated (we focus on cross-modal retrieval between language and visual representations). The latter studies how to prevent forgetting a previously learned task when learning a new one. While humans excel in these two aspects, deep neural networks are still quite limited. In this paper, we propose a combination of both problems into a continual cross-modal retrieval setting, where we study how the catastrophic interference caused by new tasks impacts the embedding spaces and their cross-modal alignment required for effective retrieval. We propose a general framework that decouples the training, indexing and querying stages. We also identify and study different factors that may lead to forgetting, and propose tools to alleviate it. We found that the indexing stage pays an important role and that simply avoiding reindexing the database with updated embedding networks can lead to significant gains. We evaluated our methods in two image-text retrieval datasets, obtaining significant gains with respect to the fine tuning baseline.
BibTeX
@inproceedings{wang2021continual,
  author = {Kai Wang and Luis Herranz and Joost van de Weijer},
  title = {{Continual Learning in Cross-Modal Retrieval}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=z185ECvNlS},
}
Figure