Continual Vision-Language Retrieval via Dynamic Knowledge Rectification

Zhenyu Cui, Yuxin Peng, Xun Wang, Manyu Zhu, Jiahuan Zhou

TL;DR

This paper introduces Dynamic Knowledge Rectification (DKR), a novel framework designed to prevent vision-language models from "forgetting" old information when learning new image-text retrieval tasks. By intelligently filtering and correcting the similarity scores (affinities) transferred from old models to new ones, DKR strikes a superior balance between retaining past knowledge and accurately acquiring new concepts.

The Challenge of Continual Vision-Language Retrieval

Large-scale vision-language models like CLIP are powerful but struggle with Continual Vision-Language Retrieval (CVRL). In real-world scenarios, data arrives in a stream—for example, moving from general objects to medical images and then to satellite photos. Fine-tuning a model on new data often leads to catastrophic forgetting, where the model loses its ability to perform well on previous datasets.

Overview of Continual Vision-Language Retrieval (CVRL) challenges, including unseen classes, combinations, and distributions within a data stream.

While "Class Incremental Learning" (CIL) exists for image classification, retrieval is more complex because it lacks fixed category labels and relies on nuanced text descriptions. Existing solutions typically use "knowledge distillation" to make the new model mimic the old one. However, the authors identify a critical flaw: if the old model calculates an incorrect affinity (similarity) for a new image-text pair, blindly distilling that "knowledge" sabotages the new model's learning process.

Methodology: Dynamic Knowledge Rectification (DKR)

DKR addresses the "incorrect affinity" problem by treating the old model as a fallible mentor. The process involves three key stages:

  1. Incorrect Knowledge Filtering: The system compares affinity matrices from the old and new models to identify instances where the old model fails to match pairs that the new model (or ground truth) handles correctly.
  2. Knowledge Rectification: Instead of discarding confused outputs, DKR fixes them using two strategies:
  3. Rectification with New Knowledge (RNK): If the new model is confident and correct where the old model failed, DKR adopts the new model's values.
  4. Rectification with Paired Knowledge (RPK): If neither model performs well, DKR uses ground-truth labels to enforce the correct relationship during distillation.
  5. Distillation: The model uses a Jensen-Shannon (JS) divergence loss to transfer this rectified knowledge matrix, ensuring stability on old tasks without inheriting the old model's errors on new data.

Experimental Setup and Results

The researchers evaluated DKR on five diverse benchmarks: MS-COCO, Flickr30K, IAPR TC-12, ECommerce-T2I (EC), and RSICD (satellite imagery). They tested both sequential dataset learning and domain-specific consistency within a single dataset.

Key Findings

  • Superior Accuracy: DKR achieved an average Recall@1 (R@1) of 43.3% on the five-dataset sequence, outperforming the previous state-of-the-art (Mod-X) at 41.9%.
  • Anti-Forgetting: DKR maintained significantly higher performance on the initial dataset (MS-COCO) even after learning four subsequent tasks compared to standard fine-tuning.
  • Positive Transfer: In some cases, DKR achieved higher results than a model trained only on a single dataset (Zero-Shot), suggesting that its rectification process facilitates the transfer of useful knowledge between domains.

Conclusion and Future Work

DKR demonstrates that distillation is a double-edged sword; success in CVRL requires protecting the "diagonal" (correct matches) of affinity matrices while filtering out erroneous legacy knowledge. While effective, the current implementation requires keeping the old model available during training, which increases memory overhead. Future research will focus on reducing computational costs and extending DKR to video-text retrieval and generative tasks.

Summary updated

Abstract

The recent large-scale pre-trained models like CLIP have aroused great concern in vision-language tasks. However, when required to match image-text data collected in a streaming manner, namely Continual Vision-Language Retrieval (CVRL), their performances are still limited due to the catastrophic forgetting of the learned old knowledge. To handle this issue, advanced methods are proposed to distill the affinity knowledge between images and texts from the old model to the new one for anti-forgetting. Unfortunately, existing approaches neglect the impact of incorrect affinity, which prevents the balance between the anti-forgetting of old knowledge and the acquisition of new knowledge. Therefore, we propose a novel framework called Dynamic Knowledge Rectification (DKR) that simultaneously achieves incorrect knowledge filtering and rectification. Specifically, we first filter the incorrect affinity knowledge calculated by the old model on the new data. Then, a knowledge rectification method is designed to rectify the incorrect affinities while preserving the correct ones. In particular, for the new data that can only be correctly retrieved by the new model, we rectify them with the corresponding new affinity to protect them from negative transfer. Additionally, for those that can not be retrieved by either the old or the new model, we introduce paired ground-truth labels to promote the acquisition of both old and new knowledge. Extensive experiments on several benchmark datasets demonstrate the effectiveness of our DKR and its superiority against state-of-the-art methods.
BibTeX
@inproceedings{cui2024continual,
  author = {Zhenyu Cui and Yuxin Peng and Xun Wang and Manyu Zhu and Jiahuan Zhou},
  title = {{Continual Vision-Language Retrieval via Dynamic Knowledge Rectification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i10.29054},
}
Figure