Adaptive Cross-Modal Prototypes for Cross-Domain Visual-Language Retrieval

Yang Liu, Qingchao Chen, Samuel AlbanieCVPR 2021 · OpenReview

Abstract

In this paper, we study the task of visual-text retrieval in the highly practical setting in which labelled visual data with paired text descriptions are available in one domain (the "source"), but only unlabelled visual data (without text descriptions) are available in the domain of interest (the "target"). We propose the ADAPTIVE CROSS-MODAL PROTOTYPES framework which seeks to enable target domain retrieval by learning cross-modal visual-text representations while minimising both uni-modal and cross-modal distribution shift across the source and target domains. Our approach is built upon two key ideas: first, we encode the inductive bias that the learned cross-modal representations should be compositional with respect to concepts in each modality—this is achieved through clustering pretrained uni-modal features across each domain and designing a careful regularisation scheme to preserve the resulting structure. Second, we employ mutual information maximisation between cross-modal representations in the source and target domains during learning—this provides a mechanism that preserves commonalities between the domains while discarding signal in each that cannot be inferred from the other. We showcase our approach for the task of cross-domain visual-text retrieval, outperforming existing approaches for both images and videos.

Introduction

Visual-language retrieval—finding the right image or video for a text query and vice versa—usually relies on massive datasets of paired content. However, real-world deployment often involves a "target" domain (e.g., medical videos or specific movies) where visual data is abundant but corresponding text descriptions are absent. This "Unsupervised Domain Adaptation" (UDA) setting presents three major hurdles: compositionality (understanding novel combinations), reporting bias (captions describing only parts of a scene), and domain shift (variations in visual and linguistic styles).

Methodology: The ACP Framework

The authors propose the ADAPTIVE CROSS-MODAL PROTOTYPES (ACP) framework to address these challenges. The architecture relies on two primary pillars:

  1. Uni-modal Compositional Keels: Instead of relying on predefined categories, the framework uses "off-the-shelf" pre-trained models to cluster visual and textual features independently. These cluster centers, termed "keels," act as static anchors of knowledge. The model ensures that learned cross-modal embeddings preserve the semantic relationships found in these keels via a KL-divergence loss.
  2. Mutual Information (MI) Maximization: To bridge the source and target domains, the model maximizes the mutual information between prototype assignments. This forces the model to focus on features predictable across both domains while discarding domain-specific noise.
The ACP framework architecture, illustrating the interaction between source and target prototypes, uni-modal keels, and the various loss functions used to align the embedding space.

Experiments and Results

The ACP framework was evaluated across diverse benchmarks, including image-text retrieval (COCO, Open Narratives) and video-text retrieval (MSRVTT, MSVD, LSMDC). In transfer tasks like MSRVTT $\rightarrow$ MSVD, ACP achieved a significant jump in performance, reaching 16.6 R@1 compared to 14.2 for the state-of-the-art baseline.

A t-SNE visualization of the baseline source-only embedding space, demonstrating the significant distribution shift between source (red) and target (blue) domains that ACP aims to mitigate.

Visualizations of the embedding space indicate that ACP produces much tighter, more distinct clusters of shared features compared to other domain adaptation methods, effectively bridging the gap shown in baseline visualizations.

Discussion and Key Insights

A primary insight of this work is the utility of "frozen" knowledge. Pre-trained uni-modal models already contain compositional knowledge; ACP leverages these as anchors (keels) so the model only needs to learn the cross-modal mapping without disrupting existing semantic structures. Furthermore, MI maximization acts as a denoising tool, encouraging the model to ignore domain-specific "style" in favor of consistent "content."

While ACP shows strong performance, a "semantic gap" remains. Future work could explore more sophisticated modeling of keel relationships or integrate larger-scale models to further enhance the static knowledge base.

Summary updated

Abstract

In this paper, we study the task of visual-text retrieval in the highly practical setting in which labelled visual data with paired text descriptions are available in one domain (the "source"), but only unlabelled visual data (without text descriptions) are available in the domain of interest (the "target"). We propose the ADAPTIVE CROSS-MODAL PROTOTYPES framework which seeks to enable target domain retrieval by learning cross-modal visual-text representations while minimising both uni-modal and cross-modal distribution shift across the source and target domains. Our approach is built upon two key ideas: first, we encode the inductive bias that the learned cross-modal representations should be compositional with respect to concepts in each modality--this is achieved through clustering pretrained uni-modal features across each domain and designing a careful regularisation scheme to preserve the resulting structure. Second, we employ mutual information maximisation between cross-modal representations in the source and target domains during learning--this provides a mechanism that preserves commonalities between the domains while discarding signal in each that cannot be inferred from the other. We showcase our approach for the task of cross-domain visual-text retrieval, outperforming existing approaches for both images and videos.
BibTeX
@inproceedings{liu2021adaptive,
  author = {Yang Liu and Qingchao Chen and Samuel Albanie},
  title = {{Adaptive Cross-Modal Prototypes for Cross-Domain Visual-Language Retrieval}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=K1ClSvLHLv},
}
Figure