MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

Haitian Wang, Ruoxi Sun, Quantong Qiu, Juntao Li, Junhui Li, Hua Chen, Jinxiong Chang, Min ZhangarXiv 2026 · arXiv

TL;DR

While modern Multimodal Embedding Models (MEMs) boast massive context windows of 32K tokens and beyond, their actual comprehension of long, interleaved data remains largely unproven. This paper introduces MMLongEmbed, the first comprehensive benchmark designed to evaluate MEMs in long-context scenarios across text, documents, and videos. The evaluation reveals a sobering reality: current state-of-the-art models rely heavily on superficial feature matching and suffer from severe positional biases when processing long-context inputs.

What Problem Does This Paper Address?

In recent years, the theoretical context windows of Multimodal Embedding Models (MEMs) have expanded dramatically, enabling applications like multimodal Retrieval-Augmented Generation (RAG) and video search. However, a larger context window does not automatically guarantee that a model effectively processes or retains deep semantic information across long inputs.

Existing embedding benchmarks suffer from three major flaws:

  1. Limited Context Diversity: They are mostly restricted to short, text-only inputs.
  2. Lack of Length Stratification: Input length is rarely treated as a controlled variable, making it impossible to see exactly where a model's performance degrades.
  3. Insufficient Task Difficulty: Candidate pools are often inflated with random data, allowing models to succeed using shallow keyword or feature matching rather than true comprehension.

To bridge this gap, the authors present MMLongEmbed to systematically stress-test MEMs under controlled, long-context multimodal conditions.

Methods

MMLongEmbed introduces a rigorous evaluation framework built on two core design principles:

  • Length-Stratified Pooling: The benchmark partitions query-candidate pairs into distinct length intervals (ranging up to 32K tokens) while keeping candidate density constant. This isolates input length as an independent variable.
  • High-Quality Hard Distractors: To prevent models from taking shortcuts via shallow feature matching, the authors engineered targeted "hard" distractors for each task type:
  • Visual Grounding: Contrastive "needle" pairs with identical syntactic structures but different core entities (e.g., swapping "Moon" with "Venus").
  • Multi-Source Reasoning: Removing critical evidence segments from an otherwise topically similar document to break the reasoning chain.
  • Logical Summarization: Replacing large chunks of text with query-irrelevant content from topically similar documents.
  • Temporal Summarization: Shuffling video frames at semantic boundaries (e.g., chronological reversal) to disrupt the global narrative while preserving local visual features.
Figure 2: Overview of the MMLongEmbed data construction pipeline. For synthetic visual grounding, we inject target instances (“needles”) across varying insertion depths and expanding context lengths, with each depth–length configuration evaluated independently. For real-world scenarios, we construct parallel hard pools utilizing targeted distractors to effectively prevent models from exploiting superficial semantic matching shortcuts. Furthermore, these real-world tasks are strictly stratified into discrete length intervals (8K, 16K, and 32K). Note: “Temporal Summary” and “Logical Summary” are abbreviated labels in the figure due to space constraints; the full task names are Temporal Summarization and Logical Summarization, respectively.

Datasets

MMLongEmbed repurposes eight public datasets originally designed for generative tasks into retrieval problems, spanning four distinct tasks:

  1. Visual Grounding (Synthetic): Sourced from NeedleBench and MM-NIAH, using haystacks from the OBELICS dataset.
  2. Multi-Source Reasoning (Real-world): Sourced from MMDocRAG (interleaved image-text documents) and HotpotQA (text-only).
  3. Logical Summarization (Real-world): Sourced from MMLongBench (document images) and GovReport (text-only).
  4. Temporal Summarization (Real-world): Sourced from ActivityNet Captions and YoukuDenseCaption (video frame sequences).
Figure 1: Length Distribution of Candidates in the MM- LongEmbed Benchmark.

Key Insights and Ideas

  • Scale Does Not Guarantee Capability: Increasing model size or embedding dimensionality yields diminishing returns. For instance, the 8B parameter version of Qwen3-VL only marginally outperformed its 2B counterpart overall and actually performed worse on the hard subset of the MMDocRAG dataset.
  • The "Lost in the Middle" Phenomenon is Real: In synthetic visual grounding tasks, models exhibit a severe positional bias. They excel at retrieving information located at the very beginning or very end of a sequence but suffer massive performance drops when the target information is buried in the middle.
  • Rapid Degradation Past 2K Tokens: For fine-grained retrieval, most models maintain stable performance only up to roughly 2,000 tokens, after which their discriminative power continuously deteriorates.
  • Global vs. Fine-Grained Matching: Models perform relatively well on summarization tasks because they can rely on global semantic matching. However, when forced to distinguish a true video sequence from a temporally shuffled one, their performance collapses, proving they struggle to capture long-range structural and temporal dependencies.
Figure 3: Precision@5 on the Visual Grounding task across varying insertion depths and context lengths.

Key Results

The authors benchmarked 11 leading MEMs (10 open-source and 1 proprietary).

  • Top Performer: Rzen-v2-7B achieved the highest overall performance with an average Precision@1 (P@1) of 58.61%, followed closely by Ops-MM-v1-7B (53.90%) and Qwen3-VL-8B (53.41%).
  • The Easy vs. Hard Gap: While many models scored above 80% P@1 on "Easy" summarization and reasoning tasks, their performance plummeted on "Hard" subsets containing targeted distractors. For example, on Temporal Summarization (Hard), most models scored between 10% and 30% P@1, highlighting their fragility when superficial cues are removed.
Quantitative comparison of model performance across context lengths (8K, 16K, 32K)

Limitations and Future Work

While MMLongEmbed provides a vital diagnostic tool, the authors note several limitations:

  • Context Cap: The benchmark currently caps evaluations at 32K tokens, which may need to scale as models push toward million-token windows.
  • Modality Constraints: The benchmark focuses exclusively on text, image, and video inputs, omitting other modalities like audio or 3D spatial data.
  • Language Bias: All datasets within the benchmark are currently English-only, leaving the evaluation of multilingual long-context multimodal understanding for future work.

Summary updated

Abstract

Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs). However, larger context windows do not necessarily translate into effective comprehension and representation of long-context multimodal inputs, which remains a critical bottleneck for real-world deployment. To address the lack of systematic evaluation in this setting, we introduce MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios. MMLongEmbed comprises four retrieval tasks spanning multiple context-length ranges, covering text, document, and video modalities. Through extensive evaluation of state-of-the-art models, we find that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies. We further observe that performance degradation varies systematically with context length and key information placement. Moreover, models exhibit substantially different robustness to redundant contextual information across modalities. For reproducibility, the benchmark and code are publicly available.
BibTeX
@misc{wang2026mmlongembed,
  author = {Haitian Wang and Ruoxi Sun and Quantong Qiu and Juntao Li and Junhui Li and Hua Chen and Jinxiong Chang and Min Zhang},
  title = {{MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios}},
  year = {2026},
  eprint = {2606.14747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.14747},
}

Loading related content...

Figure