MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence

Yue Feng, Jinwei Hu, Qijia Lu, Jiawei Niu, Li Tan, Shuo Yuan, Ziyi Yan, Yizhen Jia, Qingzhi He, Shiping Ge, Ethan Q. Chen, Wentong Li, Limin Wang, Jie Qin

TL;DR

This paper introduces MUVR, a new benchmark and task for multi-modal untrimmed video retrieval, designed to better reflect real-world long-video platforms. MUVR addresses limitations in existing benchmarks by supporting diverse query types, a multi-level visual correspondence framework, and comprehensive evaluation metrics, revealing significant challenges for current retrieval models and large language models.

What Problem Does This Paper Address?

The explosion of video content on platforms like YouTube and Bilibili necessitates efficient video retrieval. However, existing benchmarks often focus on trimmed videos and simple text queries, failing to capture the complexity of real-world scenarios. Users on these platforms often need to find specific segments within long, untrimmed videos using a combination of text descriptions, example videos, and even tags. Current retrieval methods struggle with this "one-to-many" retrieval paradigm, diverse query modalities, and the nuanced visual understanding required for fine-grained matching across various video categories. MUVR aims to bridge this gap by providing a more realistic and challenging benchmark.

Methods

MUVR introduces a novel multi-modal untrimmed video retrieval task with a focus on a practical retrieval paradigm. This paradigm supports video-centric multi-modal queries, allowing users to express their needs through:

  • Long text descriptions: Detailed explanations of what they're looking for.
  • Video tag prompts: Keywords to filter or refine results.
  • Mask prompts: (Introduced later for MLLMs) Guiding attention to specific regions.

A key innovation is the concept of multi-level visual correspondence, which defines six levels of matching criteria (copy, event, scene, instance, action, others) to capture diverse user interests. This framework is used to organize the benchmark into five partitions: news, region, instance, dance, and others, covering a wide range of video categories. For evaluating large language models (LLMs), a Reranking Score is proposed to assess their ability to refine initial retrieval results.

Datasets

The paper introduces the MUVR benchmark, which consists of:

  • 53,462 untrimmed videos collected from the video platform Bilibili.
  • 1,050 multi-modal queries, each with text descriptions, tag prompts, and video queries.
  • 84,000 labeled positive matches.

MUVR is further divided into:

  • MUVR-Base: For evaluating retrieval models.
  • MUVR-Filter: Extends MUVR-Base with 74,000 multi-labeled tags to enable tag-based filtering.
  • MUVR-QA: A subset of challenging samples for evaluating LLMs in a question-answering format, comprising 200 discrimination questions.

This benchmark is designed to be more diverse and representative of recent video content compared to older datasets.

Key Insights and Ideas

  • Untrimmed videos and multi-modal queries are the future: MUVR strongly suggests that current models are not well-equipped to handle the complexities of real-world video retrieval, which heavily relies on untrimmed content and diverse query types.
  • Video queries are often more powerful than text alone: For many retrieval tasks, especially those involving subtle visual details, video queries provide richer information than text descriptions.
  • Multi-modal queries offer complementary benefits: Combining text and video queries generally leads to better performance, highlighting the need for effective fusion strategies.
  • Current LLMs struggle with multi-video understanding and reranking: Despite their advancements, LLMs show significant limitations in efficiently comparing and reranking multiple videos, especially when dealing with complex tag prompts.
  • Fine-grained visual correspondence is crucial: The multi-level approach to visual matching is essential for accurately categorizing and retrieving relevant content across diverse video types.

Key Results

  • EVA-CLIP, a state-of-the-art vision-language model, achieved only 58% mAP on MUVR-Base and 34% mAP on MUVR-Filter, indicating significant room for improvement.
  • Video retrieval models like InternVideo2 and S2VS performed better with pure video queries than pure text queries, especially for temporal tasks like dance.
  • Image-based VLMs like EVA-CLIP excelled with pure text queries, likely due to their training on large image-text datasets.
  • MLLMs showed accuracy above 70% on MUVR-QA questions without tag prompts, but performance dropped substantially when tag prompts were introduced.
  • Multi-frame processing for MLLMs improved performance but significantly increased inference latency.

Limitations and Future Work

  • Annotation noise: Despite rigorous annotation, minor errors may exist in MUVR.
  • Limited modalities: MUVR currently focuses on visual and textual modalities, excluding audio, which could further enrich retrieval.
  • MLLM efficiency: Current LLMs are too slow for real-time retrieval/reranking. Future work should focus on token compression and more efficient multi-video understanding.
  • Tag integration: Developing better methods for integrating tag semantics into queries and handling complex tag combinations is crucial.
  • Temporal modeling: Improving temporal modeling for dynamic actions in videos remains a challenge.
  • Fusion strategies: More effective methods for fusing information from multimodal queries are needed.

Summary updated

Abstract

We propose the Multi-modal Untrimmed Video Retrieval task, along with a new benchmark (MUVR) to advance video retrieval for long-video platforms. MUVR aims to retrieve untrimmed videos containing relevant segments using multi-modal queries. It has the following features: **1) Practical retrieval paradigm:** MUVR supports video-centric multi-modal queries, expressing fine-grained retrieval needs through long text descriptions, video tag prompts, and mask prompts. It adopts a one-to-many retrieval paradigm and focuses on untrimmed videos, tailored for long-video platform applications. **2) Multi-level visual correspondence:** To cover common video categories (e.g., news, travel, dance) and precisely define retrieval matching criteria, we construct multi-level visual correspondence based on core video content (e.g., news events, travel locations, dance moves) which users are interested in and want to retrieve. It covers six levels: copy, event, scene, instance, action, and others. **3) Comprehensive evaluation criteria:** We develop 3 versions of MUVR (i.e., Base, Filter, QA). MUVR-Base/Filter evaluates retrieval models, while MUVR-QA assesses MLLMs in a question-answering format. We also propose a Reranking Score to evaluate the reranking ability of MLLMs. MUVR consists of 53K untrimmed videos from the video platform Bilibili, with 1,050 multi-modal queries and 84K matches. Extensive evaluations of 3 state-of-the-art video retrieval models, 6 image-based VLMs, and 10 MLLMs are conducted. MUVR reveals the limitations of retrieval methods in processing untrimmed videos and multi-modal queries, as well as MLLMs in multi-video understanding and reranking. Our code and benchmark is available at https://github.com/debby-0527/MUVR.
BibTeX
@misc{feng2025muvr,
  author = {Yue Feng and Jinwei Hu and Qijia Lu and Jiawei Niu and Li Tan and Shuo Yuan and Ziyi Yan and Yizhen Jia and Qingzhi He and Shiping Ge and Ethan Q. Chen and Wentong Li and Limin Wang and Jie Qin},
  title = {{MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence}},
  year = {2025},
  eprint = {2510.21406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.21406},
}
Figure