Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimization

Zhicheng Wang, Chen Ju, Xu Chen, Shuai Xiao, Jinsong Lan, Xiaoyong Zhu, Zhiguo CaoSubmission to ICLR 2026 · Rejected

TL;DR

Current multimodal embedding models collapse complex image-text inputs into a single embedding vector, bottlenecking rich multimodal semantics. The Parallel Decoupling Framework (PDF) resolves this by conditioning a shared Multimodal Large Language Model (MLLM) on distinct, layer-wise prefixes to roll out parallel embedding paths decorrelated via Mutual Information Minimization (MIM). At inference, PDF bypasses the multi-path structure to run via a single forward pass—delivering large performance and sample-efficiency gains with essentially zero extra computational overhead.

What Problem Does This Paper Address?

State-of-the-art multimodal representation models typically follow the SSC paradigm: mapping a Single input to a Singular embedding supervised by Contrastive loss. While inherited from earlier dual-encoder architectures (like CLIP), this setup creates two major hurdles for modern MLLMs:

  1. Architectural Mismatch: It underutilizes the prompt steerability of MLLMs—their distinct ability to produce varied, instruction-guided internal states.
  2. Information Bottleneck: Compressing multifaceted visual-textual inputs into a single vector causes severe information loss and harms out-of-distribution robustness.

Methods

PDF introduces an SPP paradigm (Single input, Parallel paths, Parallel outputs) guided by an adversarial information-theoretic objective:

  • Deep Prefix Injection: Instead of prepending input tokens, PDF injects path-specific, learnable prefix parameters directly into the key-value projections across every transformer layer of the MLLM backbone, creating $N$ parallel computational routes (the authors default to $N=2$).
  • Mutual Information Minimization (MIM): Simply initializing distinct prefixes is insufficient; the shared backbone and contrastive loss naturally drive parallel paths toward redundant representations. To prevent collapse, the framework actively minimizes the mutual information between path embeddings using the variational Contrastive Log-ratio Upper-Bound (vCLUB) estimator in a two-stage minimax game.
  • Dual-Objective Supervision: To ensure decorrelation does not produce random noise, each parallel path receives an individual InfoNCE contrastive loss alongside a contrastive loss on an aggregated representation (computed via a lightweight MLP-softmax module).
  • Zero-Overhead Single-Path Inference: During evaluation, the parallel routing and aggregation layers are discarded. A single predefined prefix path is executed, keeping inference cost virtually identical to standard baselines (+0.06% FLOPs).
                      ┌─► Path 1 (Prefix 1) ─► h_1 ┐
                      │                            ├─► MIM Loss (Decorrelate)
Input (Image + Text) ─┤                            ├─► Aggregation ─► InfoNCE Loss
                      │                            │
                      └─► Path 2 (Prefix 2) ─► h_2 ┘

Datasets

The authors train and evaluate PDF across established multimodal retrieval benchmarks:

  • MMEB Benchmark: 36 tasks total (20 in-distribution datasets spanning 662K pairs across Classification, VQA, Retrieval, and Visual Grounding; 16 out-of-distribution test sets).
  • Zero-Shot Transfer Datasets: Flickr30K, ShareGPT4V, Urban1K, Fashion200K, and InfoSeek.
  • No new datasets or benchmarks were introduced.

Key Insights and Ideas

  • Diversity as Training Regularization, Not Ensemble: Attention maps reveal that parallel paths converge on identical core semantics while maintaining subtle, distributed variations in attention weights across heads. As a result, aggregating paths at test time yields identical performance to running just a single path—confirming that MIM functions as a training-time regularizer for the shared model backbone.
  • Preventing Representational Collapse: Tracking prefix cosine similarity during training shows that without the MIM penalty, parallel paths collapse within hundreds of steps. Enforcing independence forces the model to explore a broader representational space.
  • Accelerated Convergence: Because the parallel paths force the shared weights to explore multifaceted features faster, the model requires significantly fewer steps to surpass standard baselines.

Key Results

  • MMEB Benchmark Gains: On top of the VLM2Vec-Qwen2VL baseline, PDF achieves +4.2 overall points on the 2B model (59.7 $\to$ 63.9) and +3.1 on the 7B model (65.5 $\to$ 68.6). On VLM2Vec-LLaVA-1.6-LR (7B), PDF improves performance by +8.9 points.
  • Zero-Shot Retrieval: Yields large gains on out-of-domain transfer, notably improving Recall@1 on Urban1K by +17.2 points for the Qwen2VL-2B backbone.
  • Training and Compute Efficiency: PDF-Qwen2VL-2B surpasses the fully converged baseline (59.7) in just 500 iterations (reaching 62.3), halving the overall training budget required ($500 \times 2$ paths vs. $2000 \times 1$ path).
  • Inference Footprint: PDF (Single Prefix) runs at 18.937 TFLOPs compared to baseline VLM2Vec's 18.925 TFLOPs, validating that performance boosts come at zero practical deployment latency.

Limitations and Future Work

  • Combinatorial Scaling of Paths: Increasing parallel paths beyond $N=4$ degrades accuracy (e.g., $N=8$ degrades OOD scores to 53.0 vs. 53.3). The authors cite the quadratic growth of pairwise MI terms ($C(N, 2) = 28$ pairs for $N=8$), which destabilizes the minimax optimization.
  • Future Directions: Developing more scalable, hierarchical, or non-pairwise mutual information objectives to accommodate larger sets of parallel paths without training instability.

Summary updated

Abstract

Embedding models are a cornerstone of modern AI. Driven by Multimodal Large Language Models (MLLMs), they have made great progress in architecture and data curation, while the holistic paradigm is still limited to SSC, {\em i.e.}, single input, singular embedding, contrastive supervision, which collapses rich, multifaceted inputs into monolithic embeddings and fails to fully exploit MLLM capabilities. In this paper, we tailor one \textbf{P}arallel \textbf{D}ecoupling \textbf{F}ramework (PDF) for multimodal embedding learning, by utilizing the proprietary steerability of MLLMs, {\em i.e.}, their ability to flexibly generate quite differentiated response under explicit instructions. Concretely, PDF conditions a shared MLLM backbone on distinct, learnable prefixes to roll out multiple parallel paths for one input, then relies on these paths to obtain parallel embeddings. To promote full parallel diversity, we employ Mutual Information Minimization (MIM) as an explicit constraint, coupled with per-path contrastive supervision to maintain semantic alignment. Such dual-objectives force PDF to yield robust semantic coverage and a generalizable embedding space. Ultimately, the remarkable embedding space are accessible at inference via one single forward pass, incurring negligible computational overhead. We instantiate PDF on multiple MLLM backbones and prove its effectiveness on MMEB benchmark. Significant gains are consistently achieved across various resolutions and model sizes, {\em e.g.}, boosting the VLM2Vec-LLaVA-1.6-LR model by a remarkable +8.9\% (7B), and the VLM2Vec-Qwen2VL models by +4.2\% (2B) and +3.1\% (7B). In terms of efficiency, our 2B model surpasses its baseline by +2.6\% using only half the computational budget. Code will be available.
BibTeX
@misc{wang2025explore,
  author = {Zhicheng Wang and Chen Ju and Xu Chen and Shuai Xiao and Jinsong Lan and Xiaoyong Zhu and Zhiguo Cao},
  title = {{Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimization}},
  year = {2025},
  eprint = {2511.01588},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.01588},
}
Figure