MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval

Qiyu Wu, Shuyang Cui, Satoshi Hayakawa, Wei-Yao Wang, Hiromi Wakaki, Yuki MitsufujiSubmission to ICLR 2026 · Withdrawn

TL;DR

When using unified Multimodal Large Language Models (MLLMs) as embedding models for composed retrieval (e.g., image + modifying text), standard contrastive learning often causes models to rely on "modality shortcuts" by ignoring one modality entirely. To fix this, the authors introduce Modality Composition Awareness (MCA), a framework combining a preference loss and a compositional prototype regularization loss. MCA significantly improves out-of-distribution (OOD) retrieval and zero-shot visual grounding without harming in-domain performance.

What Problem Does This Paper Address?

Traditional multimodal retrieval systems (like CLIP) use dual, separate encoders for text and images, requiring external fusion layers when dealing with composed queries (e.g., an image of a red dress plus the instruction "make it blue"). Modern MLLMs can natively ingest multiple modalities within a single unified architecture.

However, fine-tuning unified MLLMs using standard contrastive learning (CL) creates an unintended side effect: modality shortcut learning. Because the model can minimize the contrastive loss by relying disproportionately on whichever modality is easiest or most dominant (e.g., matching visually similar images while ignoring text instructions, or vice-versa), the learned composed representations collapse toward unimodal embeddings. Consequently, these models suffer severe performance drops under distribution shifts or out-of-domain tasks.

Methods

The authors propose Modality Composition Awareness (MCA), which supplements the standard InfoNCE contrastive loss ($\mathcal{L}_{\text{CL}}$) with two structural objectives whenever an input is multimodal:

$$\mathcal{L} = \mathcal{L}_{\text{CL}} + \alpha \mathcal{L}_{\text{MCP}} + \beta \mathcal{L}_{\text{MCR}}$$

                      ┌──────────────────────────────────────────────┐
                      │    Unified Encoder  f_theta(x)               │
                      │                                              │
Text + Image (x) ────►│ Composed Rep. ─────────┬───────────────────┐ │
                      │                        │ (MCP)             │ │ (MCR)
                      │                        ▼                   │ │   ▼
Text-only (x_text) ──►│ Unimodal Reps. ──► Preference:             │ │ Anchored to
Image-only (x_img) ──►│                    sim(x, y+) > sim(x_m, y+)│ │ mix(x_text, x_img)
                      └────────────────────────────────────────────┴─┘
  1. Modality Composition Preference (MCP): Explicitly enforces that the similarity between the multimodal query $x$ and the positive target $y^+$ is strictly higher than the similarity between any of its constituent unimodal components ($x_m \in \mathcal{U}(x)$) and that same target: $$\mathcal{L}_{\text{MCP}} = \mathbb{E}_{x, y^+ \sim \mathcal{D}} \left[ - \sum_{x_m \in \mathcal{U}(x)} \log \frac{e^{\text{sim}_\theta(x, y^+)}}{e^{\text{sim}_\theta(x_m, y^+)}} \right]$$ This penalizes models that take a unimodal "shortcut" to match the target.

  2. Modality Composition Regularization (MCR): Prevents the unified multimodal representation from drifting arbitrarily far from the semantic space spanned by its individual parts. It constructs a compositional prototype using an aggregation mixer (defaulting to a lightweight gated fusion module, $\text{mix}_\phi$) and uses a contrastive objective pulling the composed embedding closer to its prototype than to negative prototypes in the batch: $$\mathcal{L}_{\text{MCR}} = \mathbb{E}_{x, X \sim \mathcal{D}} \left[ - \log \frac{e^{\text{sim}^{\text{mix}}_{\theta,\phi}(x, \mathcal{U}(x))}}{e^{\text{sim}^{\text{mix}}_{\theta,\phi}(x, \mathcal{U}(x))} + \sum_{x' \in X} e^{\text{sim}^{\text{mix}}_{\theta,\phi}(x, \mathcal{U}(x'))}} \right]$$

Datasets

The model backbone is Qwen2-VL-2B-Instruct fine-tuned via LoRA (rank 8).

  • Training Data: Six public datasets from the MMEB benchmark covering both cross-modal and composed tasks: MSCOCO, VisualNews, VisDial, CIRR, NIGHTS, and WebQA.
  • In-Domain (IND) Evaluation: The test splits of the six training datasets.
  • Out-of-Domain (OOD) Evaluation:
  • OOD Retrieval: OVEN, FashionIQ, and EDIS (assessing robustness under domain and composition shifts).
  • OOD Zero-Shot Grounding: MSCOCO-Grounding, Visual7W-Pointing, RefCOCO, and RefCOCO-Matching (assessing cross-task generalization).

Key Insights and Ideas

  • Complementarity of MCP and MCR: MCP prevents shortcutting to unimodal parts, while MCR ensures structural grounding. Ablations show that while applying either MCP or MCR alone yields modest gains (+1.0% to +1.8%), combining them triggers a synergy (+5.5% overall OOD improvement).
  • Shortcut vs. Input Resolution Trade-off: The benefits of MCA are most pronounced when visual quality is degraded. Under low input image resolution ($128 \times 128$), visual signals are ambiguous, making the model especially prone to modality shortcuts; here, stronger MCA regularization delivers an impressive +13.1% relative boost in OOD retrieval.
  • Preserved In-Domain Performance: Unlike many regularizers that trigger severe trade-offs, MCA maintains identical in-domain accuracy (73.2% vs. 73.2% on high resolution), acting purely as a guardrail against shortcut generalization failures.
  • Mixer Flexibility: Simple non-parametric mean pooling underperforms, but learnable lightweight aggregators like Gated Fusion and Multimodal Factorized Bilinear Pooling (MFB) perform robustly, proving that the concept of prototype consistency matters more than complex fusion architecture.

Key Results

Under the default high-resolution ($1344 \times 1344$) setting:

Method In-Domain Avg. OOD All Avg. OOD Retrieval Avg. Zero-Shot Grounding Avg.
Vanilla CL 73.2% 56.3% 54.2% 57.9%
CL + MCP 73.3% 56.9% (+1.0%) 55.0% (+1.5%) 58.3% (+0.7%)
CL + MCR 73.2% 57.3% (+1.8%) 55.4% (+2.2%) 58.8% (+1.6%)
MCA (Full) 73.2% 59.4% (+5.5%) 57.4% (+5.9%) 61.0% (+5.4%)
  • t-SNE Visualizations: Vanilla CL causes composed queries to bleed heavily into text-only query clusters. MCA results in distinct, well-separated representation boundaries between multimodal and unimodal embeddings.
  • Qualitative Observations: In composed retrieval tasks (e.g., retrieving an image based on an existing photo and the instruction "wearing a blue collar"), the vanilla baseline consistently ignored modification text or image grounding cues; MCA successfully resolved both constraints simultaneously.

Limitations and Future Work

  • Assumes Shortcuts Only in Multimodal Inputs: MCA's objectives activate only when at least one input in a pair is composed, overlooking potential shortcuts that might linger in purely unimodal cross-modal queries.
  • Hyperparameter Sensitivity under Noise: Under extremely low resolution and very small loss weights ($\alpha = \beta = 0.01$), MCA caused training instability and underperformed vanilla CL, showing sensitivity to hyperparameter tuning when inputs are noisy.
  • Limited to Vision + Text: While formulated generally, the experiments only evaluate image and text composition due to current dataset constraints. Validation across audio, video, or 3+ simultaneous modalities remains open for future research.

Summary updated

Abstract

Multimodal retrieval, which seeks to retrieve relevant content across modalities such as text or image, supports applications from AI search to contents production. Despite the success of separate-encoder approaches like CLIP align modality-specific embeddings with contrastive learning, recent multimodal large language models (MLLMs) enable a unified encoder that directly processes composed inputs. While flexible and advanced, we identify that unified encoders trained with conventional contrastive learning are prone to learn modality shortcut, leading to poor robustness under distribution shifts. We propose a modality composition awareness framework to mitigate this issue. Concretely, a preference loss enforces multimodal embeddings to outperform their unimodal counterparts, while a composition regularization objective aligns multimodal embeddings with prototypes composed from its unimodal parts. These objectives explicitly model structural relationships between the composed representation and its unimodal counterparts. Experiments on various benchmarks show gains in out-of-distribution retrieval, highlighting modality composition awareness as a effective principle for robust composed multimodal retrieval when utilizing MLLMs as the unified encoder.
BibTeX
@misc{wu2025mca,
  author = {Qiyu Wu and Shuyang Cui and Satoshi Hayakawa and Wei-Yao Wang and Hiromi Wakaki and Yuki Mitsufuji},
  title = {{MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval}},
  year = {2025},
  eprint = {2510.15543},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.15543},
}

Loading related content...

Figure