Introduction
Cross-modal retrieval—the task of finding images or videos using text queries—is a cornerstone of modern AI. However, achieving high performance in "low-resource" languages remains a significant hurdle. While English-centric models benefit from millions of human-labeled pairs, most of the world's 6,000+ languages lack such datasets. Cross-lingual Cross-modal Retrieval (CCR) addresses this by leveraging machine translation (MT) to transfer knowledge from a source language (e.g., English) to a target language.
The Challenge: Linguistic Diversity and Static Adapters
A common strategy in CCR involves using "adapters"—small, trainable modules inserted into frozen, pre-trained vision-language models like CLIP. However, traditional adapters are typically static, meaning they apply the same parameters regardless of the input sentence's structure or style. This is problematic because language is inherently diverse.
As shown in the figure above, a single image can be described in numerous ways (e.g., "A cat sitting on the ground" vs. "On the ground, there is a cat"). Static adapters often fail to account for these nuances, leading to noisy representations and suboptimal retrieval performance.
The DASD Framework: Dynamic Semantics Disentangling
To overcome these limitations, the researchers proposed Dynamic Adapter with Semantics Disentangling (DASD). Unlike static methods, DASD generates unique adapter parameters "on-the-fly" for every input caption, allowing the model to adapt to specific linguistic styles.
The framework consists of three core components:
- Frozen Backbone: A pre-trained vision-language model (e.g., CLIP) is kept frozen to preserve its foundational knowledge and prevent "forgetting."
- Semantics Disentangling Module: This module decomposes the input caption into two distinct feature sets: Semantic-related features ($f^{sr}$), capturing the core meaning (actions, participants), and Semantic-agnostic features ($f^{sa}$), capturing structural style (length, word order).
- Dynamic Parameter Generation: These disentangled features are fed into a parameter generator that calculates a custom weight matrix ($W^z$) for the adapters inserted into the model's layers.
To ensure effective separation, the model employs adversarial training. A discriminator attempts to predict semantic content from style features; the model is trained to "fool" this discriminator, forcing the style features to be truly independent of the core meaning.
Experimental Evaluation and Results
DASD was evaluated across several major benchmarks, including MSCOCO (image-text) and MSRVTT (video-text), covering languages such as Chinese, Japanese, German, French, and Czech.
- State-of-the-Art Performance: On the MSCOCO (Chinese) dataset, DASD achieved a mean Average Recall (mAR) of 88.5, significantly outperforming previous leading models like UC2 (82.0) and MLA (80.4).
- Cross-lingual Robustness: In video retrieval (MSRVTT), DASD showed a substantial improvement in the "Sum of Recalls" across eight tested languages, rising from 170.2 to 206.1.
- Efficiency: Despite its dynamic nature, DASD remains parameter-efficient. It utilizes only 134M trainable parameters, which is comparable to static methods but far less than full-model fine-tuning approaches.
Conclusion and Future Outlook
DASD represents a significant step forward in cross-lingual retrieval. By moving from static to dynamic adapters and explicitly disentangling semantics from style, the model handles the inherent noise and diversity of machine-translated data more effectively. While the model still relies on machine translation for initial training signals, its "plug-and-play" approach paves the way for more inclusive AI that performs reliably across a wide array of global languages without requiring massive human-annotated datasets.
Summary updated