Abstract
Multimodal retrieval, which seeks to retrieve relevant content across modalities such as text or image, supports applications from AI search to contents production. Despite the success of separate-encoder approaches like CLIP align modality-specific embeddings with contrastive learning, recent multimodal large language models (MLLMs) enable a unified encoder that directly processes composed inputs. While flexible and advanced, we identify that unified encoders trained with conventional contrastive learning are prone to learn modality shortcut, leading to poor robustness under distribution shifts. We propose a modality composition awareness framework to mitigate this issue. Concretely, a preference loss enforces multimodal embeddings to outperform their unimodal counterparts, while a composition regularization objective aligns multimodal embeddings with prototypes composed from its unimodal parts. These objectives explicitly model structural relationships between the composed representation and its unimodal counterparts. Experiments on various benchmarks show gains in out-of-distribution retrieval, highlighting modality composition awareness as a effective principle for robust composed multimodal retrieval when utilizing MLLMs as the unified encoder.
Introduction
Composed Multimodal Retrieval (CMR) allows users to search for content using a combination of modalities, such as an image paired with text instructions for modification. For instance, a user might provide an image of a car and specify "make it an SUV and remove the snow." While modern Multimodal Large Language Models (MLLMs) offer a unified architecture for these tasks, they are often susceptible to "modality shortcut learning." This occurs when the model relies on a single, easily accessible signal (e.g., the image) while ignoring more complex instructions (e.g., the text), leading to brittle performance in real-world scenarios and distribution shifts.
The Modality Shortcut Challenge
The "modality shortcut" problem stems from the model's tendency to take the path of least resistance during training. If the model can achieve satisfactory results by focusing on just one modality, it may fail to learn the intricate relationship between the image and the text.
As illustrated in Figure 1(a), a model taking a shortcut might retrieve an image that matches the original query image but ignores the requested changes. This behavior is reflected in the embedding space: Figure 1(b) shows that with vanilla contrastive learning, the "Composed Query" (red) remains distant from the "Target" (yellow), often clustering too closely with unimodal inputs. In contrast, Figure 1(c) demonstrates how our proposed MCA framework aligns the composed query more effectively with the target.
The MCA Framework
To address these shortcuts, we introduce the Modality Composition Awareness (MCA) framework, which incorporates two primary training objectives:
- Modality Composition Preference (MCP): This loss function explicitly enforces that the multimodal query (image + text) must be more effective at retrieving the target than either the image or text alone. This prevents the model from becoming "lazy" and relying on a single modality.
- Modality Composition Regularization (MCR): This objective ensures structural consistency by aligning the unified multimodal embedding with a "prototype" composed of its unimodal parts. The authors found that a Gated Fusion approach—which learns dynamic weights for each modality—served as the most effective mixer for this regularization.
Experimental Setup and Results
The models were trained on a large-scale union of six datasets, including MSCOCO, VisualNews, and CIRR. Evaluation focused on both In-Domain (IND) performance and Out-of-Distribution (OOD) robustness using benchmarks like OVEN (Wikipedia entity linking) and FashionIQ.
Key Findings
- Robustness Gains: MCA achieved a +5.9% improvement in OOD retrieval and a +5.4% boost in zero-shot grounding tasks.
- Resolution Sensitivity: Modality shortcuts are more prevalent when image resolution is low. MCA provides significant gains in these settings by forcing the model to utilize the complementary text signal.
- Synergistic Effects: Ablation studies confirmed that MCP and MCR are complementary; MCP discourages shortcuts, while MCR stabilizes the embedding space.
Conclusion
The MCA framework demonstrates that explicitly modeling the structural relationships between modalities is crucial for robust multimodal retrieval. By mitigating modality shortcuts, MCA enables MLLMs to function as more reliable unified encoders, particularly in challenging out-of-distribution environments. Future research will look into extending these principles to other modalities like audio and video.
Summary updated