Abstract
In this paper, we study the task of visual-text retrieval in the highly practical setting in which labelled visual data with paired text descriptions are available in one domain (the "source"), but only unlabelled visual data (without text descriptions) are available in the domain of interest (the "target"). We propose the ADAPTIVE CROSS-MODAL PROTOTYPES framework which seeks to enable target domain retrieval by learning cross-modal visual-text representations while minimising both uni-modal and cross-modal distribution shift across the source and target domains. Our approach is built upon two key ideas: first, we encode the inductive bias that the learned cross-modal representations should be compositional with respect to concepts in each modality—this is achieved through clustering pretrained uni-modal features across each domain and designing a careful regularisation scheme to preserve the resulting structure. Second, we employ mutual information maximisation between cross-modal representations in the source and target domains during learning—this provides a mechanism that preserves commonalities between the domains while discarding signal in each that cannot be inferred from the other. We showcase our approach for the task of cross-domain visual-text retrieval, outperforming existing approaches for both images and videos.
Introduction
Visual-language retrieval—finding the right image or video for a text query and vice versa—usually relies on massive datasets of paired content. However, real-world deployment often involves a "target" domain (e.g., medical videos or specific movies) where visual data is abundant but corresponding text descriptions are absent. This "Unsupervised Domain Adaptation" (UDA) setting presents three major hurdles: compositionality (understanding novel combinations), reporting bias (captions describing only parts of a scene), and domain shift (variations in visual and linguistic styles).
Methodology: The ACP Framework
The authors propose the ADAPTIVE CROSS-MODAL PROTOTYPES (ACP) framework to address these challenges. The architecture relies on two primary pillars:
- Uni-modal Compositional Keels: Instead of relying on predefined categories, the framework uses "off-the-shelf" pre-trained models to cluster visual and textual features independently. These cluster centers, termed "keels," act as static anchors of knowledge. The model ensures that learned cross-modal embeddings preserve the semantic relationships found in these keels via a KL-divergence loss.
- Mutual Information (MI) Maximization: To bridge the source and target domains, the model maximizes the mutual information between prototype assignments. This forces the model to focus on features predictable across both domains while discarding domain-specific noise.
Experiments and Results
The ACP framework was evaluated across diverse benchmarks, including image-text retrieval (COCO, Open Narratives) and video-text retrieval (MSRVTT, MSVD, LSMDC). In transfer tasks like MSRVTT $\rightarrow$ MSVD, ACP achieved a significant jump in performance, reaching 16.6 R@1 compared to 14.2 for the state-of-the-art baseline.
Visualizations of the embedding space indicate that ACP produces much tighter, more distinct clusters of shared features compared to other domain adaptation methods, effectively bridging the gap shown in baseline visualizations.
Discussion and Key Insights
A primary insight of this work is the utility of "frozen" knowledge. Pre-trained uni-modal models already contain compositional knowledge; ACP leverages these as anchors (keels) so the model only needs to learn the cross-modal mapping without disrupting existing semantic structures. Furthermore, MI maximization acts as a denoising tool, encouraging the model to ignore domain-specific "style" in favor of consistent "content."
While ACP shows strong performance, a "semantic gap" remains. Future work could explore more sophisticated modeling of keel relationships or integrate larger-scale models to further enhance the static knowledge base.
Summary updated