Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimization

Zhicheng Wang, Chen Ju, Xu Chen, Shuai Xiao, Jinsong Lan, Xiaoyong Zhu, Ying Chen, Zhiguo Cao

Abstract

Traditional Multimodal Large Language Model (MLLM) embeddings rely on a "Single input, Singular embedding" (SSC) paradigm, which often fails to capture the multifaceted semantic richness of complex inputs. We present the Parallel Decoupling Framework (PDF), a novel training paradigm that utilizes the steerability of MLLMs to generate diverse, parallel embeddings. By incorporating a Mutual Information Minimization (MIM) constraint, PDF encourages the exploration of distinct semantic subspaces during training. Experimental results on the MMEB benchmark demonstrate that PDF significantly enhances embedding quality and zero-shot generalization across various architectures, achieving state-of-the-art performance with zero additional inference overhead.

Introduction

The current standard for multimodal embedding models involves mapping a single input to a singular vector using contrastive supervision. However, this approach creates an information bottleneck, especially for high-resolution images or detailed text, where a single vector cannot represent all nuances. Furthermore, MLLMs are inherently steerable via prompts, a property underutilized in static encoding paradigms. We address these limitations by proposing a framework that "explores" the representation space more thoroughly during training to "learn" a more robust embedding.

The Parallel Decoupling Framework (PDF)

The PDF paradigm shifts from the traditional SSC approach to a "Single input, Parallel path, Parallel output" (SPP) strategy.

Comparison between the traditional SSC paradigm and the proposed PDF (SPP) paradigm.

Core Components

  1. Deep Prefix Injection: The model employs $N$ parallel paths, each steered by unique learnable prefixes injected into every transformer layer of the MLLM backbone.
  2. Mutual Information Minimization (MIM): To prevent redundancy among parallel paths, a variational upper bound (vCLUB estimator) penalizes statistical dependence between embeddings. This forces each path to capture different "facets" of the input.
  3. Dual-Objective Training: Each path is optimized using a standard Contrastive Loss (InfoNCE) alongside the MIM constraint.
  4. Efficient Inference: At deployment, only a single pre-determined prefix is used. The parallel architecture is strictly a training-time mechanism, ensuring zero added computational cost during inference.

Experimental Evaluation

We evaluated PDF using the Multimodal Embedding Benchmark (MMEB), covering 36 tasks across classification, VQA, retrieval, and grounding.

Key Results

  • Performance Gains: PDF-enhanced Qwen2VL models showed improvements of +4.2% (2B) and +3.1% (7B) on MMEB. LLaVA-1.6 (7B) saw a +8.9% boost in low-resolution settings.
  • Zero-Shot Generalization: On the Urban1K dataset, the 2B model improved by +17.2 points, showcasing exceptional transferability to unseen data.
  • Training Efficiency: The framework acts as a powerful regularizer; the 2B model reached superior performance using only 50% of the computational budget compared to the baseline.

Conclusion

The Parallel Decoupling Framework demonstrates that leveraging the steerability of MLLMs through parallel paths and mutual information minimization leads to more semantically rich and robust embeddings. By decoupling the "exploration" phase during training from the "execution" phase at inference, PDF provides a scalable and efficient path toward better multimodal representations.

Summary updated

Abstract

Embedding models are a cornerstone of modern AI. Driven by Multimodal Large Language Models (MLLMs), they have made great progress in architecture and data curation, while the holistic paradigm is still limited to SSC, {\em i.e.}, single input, singular embedding, contrastive supervision, which collapses rich, multifaceted inputs into monolithic embeddings and fails to fully exploit MLLM capabilities. In this paper, we tailor one \textbf{P}arallel \textbf{D}ecoupling \textbf{F}ramework (PDF) for multimodal embedding learning, by utilizing the proprietary steerability of MLLMs, {\em i.e.}, their ability to flexibly generate quite differentiated response under explicit instructions. Concretely, PDF conditions a shared MLLM backbone on distinct, learnable prefixes to roll out multiple parallel paths for one input, then relies on these paths to obtain parallel embeddings. To promote full parallel diversity, we employ Mutual Information Minimization (MIM) as an explicit constraint, coupled with per-path contrastive supervision to maintain semantic alignment. Such dual-objectives force PDF to yield robust semantic coverage and a generalizable embedding space. Ultimately, the remarkable embedding space are accessible at inference via one single forward pass, incurring negligible computational overhead. We instantiate PDF on multiple MLLM backbones and prove its effectiveness on MMEB benchmark. Significant gains are consistently achieved across various resolutions and model sizes, {\em e.g.}, boosting the VLM2Vec-LLaVA-1.6-LR model by a remarkable +8.9\% (7B), and the VLM2Vec-Qwen2VL models by +4.2\% (2B) and +3.1\% (7B). In terms of efficiency, our 2B model surpasses its baseline by +2.6\% using only half the computational budget. Code will be available.
BibTeX
@misc{wang2025explore,
  author = {Zhicheng Wang and Chen Ju and Xu Chen and Shuai Xiao and Jinsong Lan and Xiaoyong Zhu and Ying Chen and Zhiguo Cao},
  title = {{Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimization}},
  year = {2025},
  eprint = {2511.01588},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.01588},
}
Figure