Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval

Rui Cai, Zhiyu Dong, Jianfeng Dong, Xun Wang

Introduction

Cross-modal retrieval—the task of finding images or videos using text queries—is a cornerstone of modern AI. However, achieving high performance in "low-resource" languages remains a significant hurdle. While English-centric models benefit from millions of human-labeled pairs, most of the world's 6,000+ languages lack such datasets. Cross-lingual Cross-modal Retrieval (CCR) addresses this by leveraging machine translation (MT) to transfer knowledge from a source language (e.g., English) to a target language.

The Challenge: Linguistic Diversity and Static Adapters

A common strategy in CCR involves using "adapters"—small, trainable modules inserted into frozen, pre-trained vision-language models like CLIP. However, traditional adapters are typically static, meaning they apply the same parameters regardless of the input sentence's structure or style. This is problematic because language is inherently diverse.

Linguistic diversity in target-language captions illustrates why static adapters struggle to capture varied expressions of the same semantic content.

As shown in the figure above, a single image can be described in numerous ways (e.g., "A cat sitting on the ground" vs. "On the ground, there is a cat"). Static adapters often fail to account for these nuances, leading to noisy representations and suboptimal retrieval performance.

The DASD Framework: Dynamic Semantics Disentangling

To overcome these limitations, the researchers proposed Dynamic Adapter with Semantics Disentangling (DASD). Unlike static methods, DASD generates unique adapter parameters "on-the-fly" for every input caption, allowing the model to adapt to specific linguistic styles.

The DASD architecture features a Semantics Disentangling module and a Dynamic Parameter Generation component to create input-specific adapter weights.

The framework consists of three core components:

  1. Frozen Backbone: A pre-trained vision-language model (e.g., CLIP) is kept frozen to preserve its foundational knowledge and prevent "forgetting."
  2. Semantics Disentangling Module: This module decomposes the input caption into two distinct feature sets: Semantic-related features ($f^{sr}$), capturing the core meaning (actions, participants), and Semantic-agnostic features ($f^{sa}$), capturing structural style (length, word order).
  3. Dynamic Parameter Generation: These disentangled features are fed into a parameter generator that calculates a custom weight matrix ($W^z$) for the adapters inserted into the model's layers.

To ensure effective separation, the model employs adversarial training. A discriminator attempts to predict semantic content from style features; the model is trained to "fool" this discriminator, forcing the style features to be truly independent of the core meaning.

Experimental Evaluation and Results

DASD was evaluated across several major benchmarks, including MSCOCO (image-text) and MSRVTT (video-text), covering languages such as Chinese, Japanese, German, French, and Czech.

  • State-of-the-Art Performance: On the MSCOCO (Chinese) dataset, DASD achieved a mean Average Recall (mAR) of 88.5, significantly outperforming previous leading models like UC2 (82.0) and MLA (80.4).
  • Cross-lingual Robustness: In video retrieval (MSRVTT), DASD showed a substantial improvement in the "Sum of Recalls" across eight tested languages, rising from 170.2 to 206.1.
  • Efficiency: Despite its dynamic nature, DASD remains parameter-efficient. It utilizes only 134M trainable parameters, which is comparable to static methods but far less than full-model fine-tuning approaches.

Conclusion and Future Outlook

DASD represents a significant step forward in cross-lingual retrieval. By moving from static to dynamic adapters and explicitly disentangling semantics from style, the model handles the inherent noise and diversity of machine-translated data more effectively. While the model still relies on machine translation for initial training signals, its "plug-and-play" approach paves the way for more inclusive AI that performs reliably across a wide array of global languages without requiring massive human-annotated datasets.

Summary updated

BibTeX
@inproceedings{cai2025dynamic,
  author = {Rui Cai and Zhiyu Dong and Jianfeng Dong and Xun Wang},
  title = {{Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i2.32186},
}
Figure