U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs

Xiaojie Li, Chu Li, Shi-Zhe Chen, Xi Chen

Introduction

Universal Multimodal Retrieval (UMR) represents a significant challenge in artificial intelligence, requiring a single model to retrieve relevant candidates across diverse modalities, including text, images, and interleaved "composed" queries. While Multimodal Large Language Models (MLLMs) like LLaVA and Qwen-VL have shown immense potential, their application as embedding models remains inconsistent. Researchers often employ varied training recipes without a clear understanding of which design choices drive performance. U-MARVEL addresses this gap by systematically exploring the design space of MLLM-based embedding learning, identifying critical factors for robust retrieval performance.

Methodology

The U-MARVEL framework is built upon the Qwen2-VL-7B architecture, utilizing a multi-faceted approach to optimize embedding quality through architecture adaptation, progressive training, and efficient distillation.

The U-MARVEL framework addresses key design questions in recall and rerank model development.

Architecture Adaptation

The authors investigated the transition from causal to bidirectional attention. Unlike standard generative LLMs that use "Last Token" embeddings, U-MARVEL employs Mean Pooling combined with bidirectional attention. This allows the model to consider the entire input sequence simultaneously, effectively mitigating the recency bias common in causal models.

Progressive Training Curriculum

To handle the complexity of multimodal tasks, a three-stage training strategy was implemented:

  1. Text Retrieval Adaptation: Initial fine-tuning on Natural Language Inference (NLI) datasets to establish a strong textual baseline.
  2. Cross-modal Alignment: Aligning image and text representations using large-scale pairs (e.g., CC3M).
  3. Instruction-tuned UMR: Final training on the full suite of multimodal retrieval tasks to refine instruction-following capabilities.

Efficient Recall-then-Rerank Distillation

High-performance retrieval often relies on a computationally expensive reranker to refine initial results. U-MARVEL introduces an improved distillation method that transfers the reranker's discriminative power into the faster recall model. By focusing specifically on top-k hard negatives, this approach achieves a 26x increase in feature efficiency compared to traditional distillation methods.

Comparison between traditional and improved distillation strategies for training the recall model.

Key Insights and Design Factors

  • Bidirectional Attention and Mean Pooling: This combination significantly outperforms causal attention by providing a more holistic representation of the query, as evidenced by ablation studies.
  • Instruction Masking: A subtle but effective trick involves masking instruction tokens during mean pooling. This prevents "calculation bias," ensuring the resulting embedding focuses strictly on the query's semantic content.
  • Hard Negative Filtering: To avoid the "false negative trap"—where semantically similar but unlabeled items are incorrectly treated as negatives—U-MARVEL employs a filtering threshold to stabilize contrastive learning and prevent model collapse.
  • Learnable Temperature: Implementing a learnable temperature parameter ($\tau$) in the InfoNCE loss allows the model to adaptively sharpen its probability distribution, leading to superior performance over fixed settings.

Experimental Results

U-MARVEL was evaluated on the M-BEIR benchmark, which spans 10 datasets and 16 retrieval task types across four domains.

  • State-of-the-Art Performance: U-MARVEL achieved an average recall of 63.2% on the M-BEIR local pool, significantly surpassing the previous leader, LamRA-Ret (56.6%).
  • Zero-Shot Generalization: The model demonstrated exceptional transferability, achieving SOTA results on 9 out of 12 unseen tasks, including Flickr30K, CIRCO, and Visual Dialog.
  • Computational Efficiency: The improved distillation process was completed in just 14 hours, a drastic reduction from the estimated 340 hours required by traditional methods, without sacrificing retrieval accuracy.

Conclusion

U-MARVEL provides a comprehensive roadmap for building universal multimodal retrievers. By identifying the importance of bidirectional attention, progressive training, and efficient distillation, the authors have established a new baseline for the field. Future work will extend this framework to additional modalities like audio and explore the scaling laws of these design choices in models exceeding 70B parameters.

Summary updated

BibTeX
@inproceedings{li2026u,
  author = {Xiaojie Li and Chu Li and Shi-Zhe Chen and Xi Chen},
  title = {{U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=CUFJluq5O8},
}
Figure