AntEmbed: Dual-Mode Multimodal Embedding Model for Efficient and Fine-Grained Retrieval

Kehang Jia, Teng Liu, Juntao Li, Hua Chen, Jinxiong Chang, Xiaobo Liang, Min ZhangarXiv 2026 · arXiv

TL;DR

AntEmbed is a dual-mode multimodal embedding model that bridges the gap between lightning-fast retrieval and deep, reasoning-based understanding. By unifying discriminative and generative embeddings into a single representation space, it achieves state-of-the-art performance on multimodal benchmarks while allowing developers to choose between speed and accuracy without doubling their storage costs.

What Problem Does This Paper Address?

In the world of multimodal retrieval—finding the right image, video, or document based on a complex query—researchers usually face a trade-off between two paradigms:

  1. Discriminative Embeddings: These models (like CLIP) map inputs directly to vectors. They are incredibly efficient for large-scale online retrieval but often miss fine-grained semantic details.
  2. Generative Embeddings: These models use "Chain-of-Thought" (CoT) reasoning to analyze a query before embedding it. While much more accurate for complex tasks, they are computationally expensive and slow because they require an autoregressive generation step at inference time.

Previous attempts to combine these (like UME-R1) often treated them as two separate modes with two different embedding tokens. This resulted in "disjoint" representation spaces, meaning a system would have to maintain two separate database indexes—one for fast retrieval and one for accurate retrieval—massively increasing storage overhead. AntEmbed aims to solve this by creating a unified embedding space where both modes can coexist and even help each other.

Methods

The authors propose a two-stage training strategy to harmonize these two modes:

  • Confusion-Aware CoT Construction: To train the model, they use a powerful external reasoner to generate "reasoning trajectories." Crucially, this reasoning isn't just a description; it's "confusion-aware." The reasoner is prompted to identify specific attributes that distinguish a candidate from its five most similar "confusing" neighbors in the dataset.
  • Stage 1: Multi-positive Contrastive Learning: AntEmbed treats the original input and the reasoning-enhanced input as two semantically equivalent "views" of the same object. Using a single embedding token, the model is trained to pull both views toward the query simultaneously. This forces the discriminative and generative representations into the same neighborhood in the vector space.
  • Stage 2: Cross-Mode Self-Distillation: Here, the reasoning-enhanced branch acts as a "teacher" for the fast discriminative branch. Through similarity-distribution distillation and direct vector alignment, the model transfers the "wisdom" gained from reasoning into the fast mode. This allows the base model to perform better even when it isn't allowed to "think" (generate CoT) at inference time.
Overview of the AntEmbed two-stage training process, showing multi-positive learning and self-distillation.

Datasets

The model was trained on the MMEB-v2 training set along with other public multimodal retrieval datasets. Evaluation was conducted on:

  • MMEB-v2: A benchmark covering image, video, and visual document retrieval.
  • MMEB-v3: A newer, more challenging benchmark that introduces agent-based tasks like tool retrieval, GUI control, and long-context retrieval.

Key Insights and Ideas

  • The Unified Space Advantage: By aligning both modes, AntEmbed eliminates the need for dual indexes. As shown in Figure 2, AntEmbed significantly reduces the "Representation Space Discrepancy" compared to previous models like UME-R1.
  • Reasoning as a Training Signal: The most powerful idea here is that reasoning shouldn't just be an inference-time luxury. By using it as a distillation target, the authors improve the "base" model's performance by 4-12 points across different domains without adding a single millisecond to its inference time.
  • Distinguishing, Not Just Describing: The "confusion-aware" approach to CoT recognizes that retrieval is fundamentally about discrimination. Knowing that an image contains a "dog" is less useful than knowing it contains a "Golden Retriever" specifically to distinguish it from a "Labrador."

Key Results

AntEmbed-8B sets a new high bar for multimodal retrieval:

  • State-of-the-Art: It achieved an overall score of 81.5 on MMEB-v2, outperforming the strongest baseline (DME-9B) by 3.1 points.
  • Domain Dominance: It secured the highest scores in Image (83.9), Video (74.8), and Visual Document (82.7) retrieval.
  • Efficient Excellence: Even in its "base" mode (no reasoning at inference), AntEmbed-2B outperformed several larger generative models that did use reasoning, proving the effectiveness of the distillation process.
Figure 1: Performance Comparison on MMEB-v2. Overall and per-domain results on MMEB-v2. AntEmbed consistently outperforms competing generative and discriminative embedding models at comparable parameter scales.

Limitations and Future Work

  • Modality Gaps: The current version of AntEmbed does not support audio inputs, which meant it had to be excluded from the audio-related tasks in the MMEB-v3 benchmark.
  • Agent Integration: The authors plan to explore how agent-generated reasoning traces and broader task contexts can be integrated to better align retrieval with the evolving needs of AI agents.
  • Reasoner Dependency: The quality of the embeddings is partially dependent on the quality of the external reasoner used during training (e.g., Qwen or Gemini).

Summary updated

Abstract

Discriminative embeddings encode original multimodal inputs efficiently but struggle with fine-grained semantics. Generative embeddings use reasoning to capture query intent and relevance evidence, but incur higher computational costs. We propose AntEmbed, a dual-mode multimodal embedding model that supports efficient retrieval and reasoning-enhanced fine-grained matching in a unified embedding space. First, we construct retrieval-oriented reasoning content that captures query intent and confusion-aware semantic evidence. We treat original and reasoning-enhanced candidate inputs as semantically equivalent views, and use multi-positive contrastive learning with a single embedding token to align discriminative and generative embeddings. Second, cross-mode self-distillation transfers fine-grained semantic information from reasoning-enhanced to original representations through candidate similarity-distribution distillation and query representation alignment. This strengthens discriminative embeddings without requiring additional online reasoning. Both retrieval modes share a single candidate index. On MMEB-v2, AntEmbed-8B achieves an overall score of 81.5 with reasoning-enhanced embeddings, outperforming the strongest baseline by 3.1 points.
BibTeX
@misc{jia2026antembed,
  author = {Kehang Jia and Teng Liu and Juntao Li and Hua Chen and Jinxiong Chang and Xiaobo Liang and Min Zhang},
  title = {{AntEmbed: Dual-Mode Multimodal Embedding Model for Efficient and Fine-Grained Retrieval}},
  year = {2026},
  eprint = {2611.16405},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16405},
}
Figure