TL;DR
This paper introduces MUVR, a new benchmark and task for multi-modal untrimmed video retrieval, designed to better reflect real-world long-video platforms. MUVR addresses limitations in existing benchmarks by supporting diverse query types, a multi-level visual correspondence framework, and comprehensive evaluation metrics, revealing significant challenges for current retrieval models and large language models.
What Problem Does This Paper Address?
The explosion of video content on platforms like YouTube and Bilibili necessitates efficient video retrieval. However, existing benchmarks often focus on trimmed videos and simple text queries, failing to capture the complexity of real-world scenarios. Users on these platforms often need to find specific segments within long, untrimmed videos using a combination of text descriptions, example videos, and even tags. Current retrieval methods struggle with this "one-to-many" retrieval paradigm, diverse query modalities, and the nuanced visual understanding required for fine-grained matching across various video categories. MUVR aims to bridge this gap by providing a more realistic and challenging benchmark.
Methods
MUVR introduces a novel multi-modal untrimmed video retrieval task with a focus on a practical retrieval paradigm. This paradigm supports video-centric multi-modal queries, allowing users to express their needs through:
- Long text descriptions: Detailed explanations of what they're looking for.
- Video tag prompts: Keywords to filter or refine results.
- Mask prompts: (Introduced later for MLLMs) Guiding attention to specific regions.
A key innovation is the concept of multi-level visual correspondence, which defines six levels of matching criteria (copy, event, scene, instance, action, others) to capture diverse user interests. This framework is used to organize the benchmark into five partitions: news, region, instance, dance, and others, covering a wide range of video categories. For evaluating large language models (LLMs), a Reranking Score is proposed to assess their ability to refine initial retrieval results.
Datasets
The paper introduces the MUVR benchmark, which consists of:
- 53,462 untrimmed videos collected from the video platform Bilibili.
- 1,050 multi-modal queries, each with text descriptions, tag prompts, and video queries.
- 84,000 labeled positive matches.
MUVR is further divided into:
- MUVR-Base: For evaluating retrieval models.
- MUVR-Filter: Extends MUVR-Base with 74,000 multi-labeled tags to enable tag-based filtering.
- MUVR-QA: A subset of challenging samples for evaluating LLMs in a question-answering format, comprising 200 discrimination questions.
This benchmark is designed to be more diverse and representative of recent video content compared to older datasets.
Key Insights and Ideas
- Untrimmed videos and multi-modal queries are the future: MUVR strongly suggests that current models are not well-equipped to handle the complexities of real-world video retrieval, which heavily relies on untrimmed content and diverse query types.
- Video queries are often more powerful than text alone: For many retrieval tasks, especially those involving subtle visual details, video queries provide richer information than text descriptions.
- Multi-modal queries offer complementary benefits: Combining text and video queries generally leads to better performance, highlighting the need for effective fusion strategies.
- Current LLMs struggle with multi-video understanding and reranking: Despite their advancements, LLMs show significant limitations in efficiently comparing and reranking multiple videos, especially when dealing with complex tag prompts.
- Fine-grained visual correspondence is crucial: The multi-level approach to visual matching is essential for accurately categorizing and retrieving relevant content across diverse video types.
Key Results
- EVA-CLIP, a state-of-the-art vision-language model, achieved only 58% mAP on MUVR-Base and 34% mAP on MUVR-Filter, indicating significant room for improvement.
- Video retrieval models like InternVideo2 and S2VS performed better with pure video queries than pure text queries, especially for temporal tasks like dance.
- Image-based VLMs like EVA-CLIP excelled with pure text queries, likely due to their training on large image-text datasets.
- MLLMs showed accuracy above 70% on MUVR-QA questions without tag prompts, but performance dropped substantially when tag prompts were introduced.
- Multi-frame processing for MLLMs improved performance but significantly increased inference latency.
Limitations and Future Work
- Annotation noise: Despite rigorous annotation, minor errors may exist in MUVR.
- Limited modalities: MUVR currently focuses on visual and textual modalities, excluding audio, which could further enrich retrieval.
- MLLM efficiency: Current LLMs are too slow for real-time retrieval/reranking. Future work should focus on token compression and more efficient multi-video understanding.
- Tag integration: Developing better methods for integrating tag semantics into queries and handling complex tag combinations is crucial.
- Temporal modeling: Improving temporal modeling for dynamic actions in videos remains a challenge.
- Fusion strategies: More effective methods for fusing information from multimodal queries are needed.
Summary updated