Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval

Rui Cai, Zhiyu Dong, Jianfeng Dong, Xun Wang

Abstract

Cross-lingual Cross-modal Retrieval (CCR) aims to bridge the gap between visual content and low-resource languages by leveraging knowledge from high-resource languages like English. Traditional methods often employ static adapters within frozen pre-trained models, which fail to account for the diverse linguistic expressions and stylistic variations inherent in different languages. This paper introduces Dynamic Adapter with Semantics Disentangling (DASD), a framework that generates custom adapter parameters on-the-fly for each input caption. By disentangling core semantic meaning from stylistic, semantic-agnostic features, DASD achieves state-of-the-art performance across multiple image and video retrieval benchmarks, demonstrating superior robustness to linguistic variety and machine-translation noise.

Introduction

Cross-modal retrieval, such as searching for images using text queries, typically relies on massive human-labeled datasets. While English is well-supported, most of the world's 6,000+ languages lack sufficient data. CCR addresses this by transferring knowledge from a "rich" source language to a "poor" target language. A common approach involves using adapters—small, trainable modules—inserted into large, frozen pre-trained models like CLIP.

However, existing adapters are typically static; their weights are fixed after training. This is problematic because the same visual scene can be described using various word orders, lengths, and styles. Static adapters struggle to generalize across these variations, leading to a "language gap" that hinders retrieval accuracy.

Methodology: The DASD Framework

The core innovation of DASD is the Dynamic Parameter Generation module, which creates unique adapter weights ($W^z$) for every input sentence. This process is driven by Semantics Disentangling, which separates a caption into two distinct feature sets.

The DASD architecture featuring Dynamic Parameter Generation and cross-modal/cross-lingual alignment.

Semantics Disentangling

  1. Semantic-related features ($f^{sr}$): These capture the "what" of the sentence (objects, actions). They are optimized using a semantic consistency loss to align with the source language (English) representation.
  2. Semantic-agnostic features ($f^{sa}$): These capture the "how" (sentence length, style, grammar). These are isolated via adversarial training, ensuring they contain information that cannot be used to reconstruct the core semantic meaning.

Dynamic Parameter Generation

These two feature sets are concatenated and processed by a lightweight Multi-Layer Perceptron (MLP) to generate the weights for the adapters. These adapters are then inserted into a frozen Vision-Language Pretraining (VLP) backbone, such as CLIP or CCLM. This ensures the model retains its original knowledge while becoming highly adaptive to specific linguistic inputs.

Experimental Evaluation

The authors evaluated DASD on three major benchmarks:

  • Image-Text Retrieval: Multi30K (German, French, Czech) and MSCOCO (Chinese, Japanese).
  • Video-Text Retrieval: MSRVTT across eight languages (e.g., Russian, Spanish, Vietnamese).
  • Training Protocol: The model was trained on CC300K using machine-translated captions to simulate low-resource conditions.

DASD achieved new state-of-the-art results. For instance, on Multi30K (French), it reached a mean Average Recall (mAR) of 91.1, significantly outperforming previous methods like MLA (83.3). The framework also showed strong zero-shot performance and proved to be a "plug-and-play" solution compatible with both dual-stream (CLIP) and single-stream (CCLM) architectures.

Key Insights and Discussion

A critical finding of this work is that explicitly modeling stylistic "noise" (semantic-agnostic features) actually improves the model's ability to extract core meaning. By making the adapter input-aware, the system becomes robust to the variety found in natural and machine-translated text.

Visualization of semantic-agnostic features using t-SNE, showing natural clustering based on grammatical structure.

Visualization using t-SNE confirms that the disentangling module works as intended. The semantic-agnostic features naturally cluster based on grammatical structures—such as sentences starting with quantifiers versus those starting with locations—independently of the actual objects or actions described.

Conclusion

DASD represents a significant step forward in cross-lingual cross-modal retrieval by replacing static adapters with dynamic, input-specific ones. By disentangling semantics from style, the model effectively bridges the language gap in low-resource settings. Future work may extend this approach to more complex tasks like cross-lingual captioning or multi-modal tasks involving audio.

Summary updated

Abstract

Existing cross-modal retrieval methods typically rely on large-scale vision-language pair data. This makes it challenging to efficiently develop a cross-modal retrieval model for under-resourced languages of interest. Therefore, Cross-lingual Cross-modal Retrieval (CCR), which aims to align vision and the low-resource language (the target language) without using any human-labeled target-language data, has gained increasing attention. As a general parameter-efficient way, a common solution is to utilize adapter modules to transfer the vision-language alignment ability of Vision-Language Pretraining (VLP) models from a source language to a target language. However, these adapters are usually static once learned, making it difficult to adapt to target-language captions with varied expressions. To alleviate it, we propose Dynamic Adapter with Semantics Disentangling (DASD), whose parameters are dynamically generated conditioned on the characteristics of the input captions. Considering that the semantics and expression styles of the input caption largely influence how to encode it, we propose a semantic disentangling module to extract the semantic-related and semantic-agnostic features from the input, ensuring that generated adapters are well-suited to the characteristics of input caption. Extensive experiments on two image-text datasets and one video-text dataset demonstrate the effectiveness of our model for cross-lingual cross-modal retrieval, as well as its good compatibility with various VLP models.
BibTeX
@inproceedings{cai2025dynamic,
  author = {Rui Cai and Zhiyu Dong and Jianfeng Dong and Xun Wang},
  title = {{Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i2.32186},
}
Figure