Abstract
Extensible Cross-Modal Hashing (ECMH) is a novel framework designed to mitigate the "catastrophic forgetting" phenomenon in cross-modal retrieval systems. As real-world data categories evolve, traditional models require expensive retraining or suffer significant performance degradation on legacy data when fine-tuned. ECMH introduces a three-stage extension process—Warm-up, Code Generation, and Extending—leveraging a self-taught distillation loss and an agreement matrix to preserve existing knowledge. Our evaluation shows that ECMH reduces training time by up to 91% while maintaining full compatibility with previously generated hash codes and achieving retrieval accuracy comparable to joint training on the full dataset.
Introduction
Cross-modal retrieval enables users to search across different data types, such as using text queries to find relevant images. This is typically achieved by mapping different modalities into a shared Hamming space where similar items are represented by similar binary hash codes. However, the dynamic nature of real-world data presents a significant challenge: the arrival of new categories (e.g., "drones" or "e-scooters").
Existing solutions often fall into two extremes: retraining the entire model from scratch, which is computationally prohibitive and requires access to all historical data, or fine-tuning on new data, which leads to catastrophic forgetting. Furthermore, updating models often necessitates regenerating hash codes for the entire database, creating a massive overhead for large-scale systems. ECMH addresses these issues by providing an extensible framework that grows efficiently without compromising existing functionality.
Methodology
ECMH utilizes a dual-stream architecture comprising INet (a VGG-19 based CNN for images) and TNet (a Deep Averaging Network for text). The framework facilitates model extension through three distinct stages:
- Warm-up: The system initializes word vectors for the new vocabulary while keeping the rest of the network parameters frozen.
- Code Generation: The existing model generates "pseudo-labels" or hash codes for the new data to establish a retrieval baseline.
- Extending: This core stage updates the model using a novel objective function incorporating a Self-taught Distillation Loss ($L_d$).
The primary innovation in the extending stage is the Agreement Matrix ($A_\alpha$). During the update process, the model identifies specific dimensions in the feature space where image and text representations already exhibit high confidence and similarity (agreement). The distillation loss then acts as a protective mask, preventing the model from altering these "agreed" dimensions. This mechanism preserves old knowledge while allowing "disagreed" dimensions to adapt to the new categories.
Experimental Evaluation
The authors evaluated ECMH on two benchmark datasets, MIRFLICKR-25k and MSCOCO-2014, modified to simulate incremental growth through super-category and sub-category extensions.
Key Results
- Efficiency: ECMH achieved massive speedups, reducing training time by 91% on the Flickr dataset and 67% on MSCOCO compared to joint training (retraining on the full combined dataset).
- Stability: Unlike standard fine-tuning, which saw an 11.3% performance drop on old tasks in MSCOCO, ECMH maintained stable or slightly improved performance on legacy categories.
- Performance Trade-offs: As shown in the retrieval metrics, ECMH maintains a high Mean Average Precision (MAP) across both old and new tasks, staying close to the theoretical upper bound of joint training.
Conclusion
ECMH represents a significant step toward "evergreen" cross-modal retrieval systems. By focusing updates on new data and selectively preserving feature dimensions through self-taught distillation, the framework achieves high efficiency and prevents catastrophic forgetting. Future work may explore extending this approach to more than two modalities and handling continuous streaming data where categories arrive sequentially.
Summary updated