Abstract
Cross-lingual Cross-modal Retrieval (CCR) aims to bridge the gap between visual content and low-resource languages by leveraging knowledge from high-resource languages like English. Traditional methods often employ static adapters within frozen pre-trained models, which fail to account for the diverse linguistic expressions and stylistic variations inherent in different languages. This paper introduces Dynamic Adapter with Semantics Disentangling (DASD), a framework that generates custom adapter parameters on-the-fly for each input caption. By disentangling core semantic meaning from stylistic, semantic-agnostic features, DASD achieves state-of-the-art performance across multiple image and video retrieval benchmarks, demonstrating superior robustness to linguistic variety and machine-translation noise.
Introduction
Cross-modal retrieval, such as searching for images using text queries, typically relies on massive human-labeled datasets. While English is well-supported, most of the world's 6,000+ languages lack sufficient data. CCR addresses this by transferring knowledge from a "rich" source language to a "poor" target language. A common approach involves using adapters—small, trainable modules—inserted into large, frozen pre-trained models like CLIP.
However, existing adapters are typically static; their weights are fixed after training. This is problematic because the same visual scene can be described using various word orders, lengths, and styles. Static adapters struggle to generalize across these variations, leading to a "language gap" that hinders retrieval accuracy.
Methodology: The DASD Framework
The core innovation of DASD is the Dynamic Parameter Generation module, which creates unique adapter weights ($W^z$) for every input sentence. This process is driven by Semantics Disentangling, which separates a caption into two distinct feature sets.
Semantics Disentangling
- Semantic-related features ($f^{sr}$): These capture the "what" of the sentence (objects, actions). They are optimized using a semantic consistency loss to align with the source language (English) representation.
- Semantic-agnostic features ($f^{sa}$): These capture the "how" (sentence length, style, grammar). These are isolated via adversarial training, ensuring they contain information that cannot be used to reconstruct the core semantic meaning.
Dynamic Parameter Generation
These two feature sets are concatenated and processed by a lightweight Multi-Layer Perceptron (MLP) to generate the weights for the adapters. These adapters are then inserted into a frozen Vision-Language Pretraining (VLP) backbone, such as CLIP or CCLM. This ensures the model retains its original knowledge while becoming highly adaptive to specific linguistic inputs.
Experimental Evaluation
The authors evaluated DASD on three major benchmarks:
- Image-Text Retrieval: Multi30K (German, French, Czech) and MSCOCO (Chinese, Japanese).
- Video-Text Retrieval: MSRVTT across eight languages (e.g., Russian, Spanish, Vietnamese).
- Training Protocol: The model was trained on CC300K using machine-translated captions to simulate low-resource conditions.
DASD achieved new state-of-the-art results. For instance, on Multi30K (French), it reached a mean Average Recall (mAR) of 91.1, significantly outperforming previous methods like MLA (83.3). The framework also showed strong zero-shot performance and proved to be a "plug-and-play" solution compatible with both dual-stream (CLIP) and single-stream (CCLM) architectures.
Key Insights and Discussion
A critical finding of this work is that explicitly modeling stylistic "noise" (semantic-agnostic features) actually improves the model's ability to extract core meaning. By making the adapter input-aware, the system becomes robust to the variety found in natural and machine-translated text.
Visualization using t-SNE confirms that the disentangling module works as intended. The semantic-agnostic features naturally cluster based on grammatical structures—such as sentences starting with quantifiers versus those starting with locations—independently of the actual objects or actions described.
Conclusion
DASD represents a significant step forward in cross-lingual cross-modal retrieval by replacing static adapters with dynamic, input-specific ones. By disentangling semantics from style, the model effectively bridges the language gap in low-resource settings. Future work may extend this approach to more complex tasks like cross-lingual captioning or multi-modal tasks involving audio.
Summary updated