Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models

Yifang Xu, Yunzhuo Sun, Benxiang Zhai, Ming Li, Wenxin Liang, Yang Li, Sidan DuAAAI 2025 · DOI

Introduction

Video Moment Retrieval (VMR) is a critical task in computer vision and natural language processing, requiring a system to identify the precise start and end timestamps of an event within an untrimmed video based on a natural language query. Traditionally, achieving high performance in VMR has necessitated large-scale datasets with dense manual annotations. However, the cost and time required for such annotations have led researchers toward "zero-shot" approaches that do not require task-specific training.

A significant hurdle for existing zero-shot VMR methods is language bias. Human-generated queries are often imperfect, containing grammatical errors, typos, or brand-specific terms (e.g., "Kleenex" instead of "tissue"). These linguistic nuances can mislead models, causing them to focus on irrelevant features or fail to localize the event entirely. Furthermore, while many Multimodal Large Language Models (MLLMs) excel at static image description, they often struggle with the temporal consistency required for long-form video analysis.

Figure 1: Overview of the VMR task and the proposed Moment-GPT compared to existing paradigms. Note how language-biased words (red) can affect localization.

Methodology: The Moment-GPT Pipeline

To address these challenges, the authors introduce Moment-GPT, a tuning-free, multi-agent pipeline that orchestrates several off-the-shelf MLLMs to perform robust zero-shot VMR. The architecture is modular, leveraging the specialized strengths of different frozen models in a hierarchical fashion.

1. Query Debiasing

The first stage involves using LLaMA-3 to preprocess the user's query. By treating the query as a text-to-text task, the system rectifies grammatical errors and generates multiple diverse, simplified versions of the original prompt. This "debiasing" ensures that subsequent visual models are not confused by non-standard language, focusing instead on the core semantic meaning.

2. Adaptive Candidate Generation

Rather than relying on a rigid sliding window approach, Moment-GPT generates candidate video segments (spans) adaptively:

  • Frame Captioning: MiniGPT-v2 generates detailed textual descriptions for every frame in the video.
  • Frame Scoring: LLaMA-3 calculates the semantic similarity between these frame-level captions and the debiased queries.
  • Span Generation: Using an inverse cumulative histogram of these scores, the system dynamically groups high-similarity frames into potential event segments.

3. Hierarchical Span Selection

To capture the temporal context that individual frames lack, the system performs a second level of analysis:

  • Span Summarization: Video-ChatGPT provides a holistic summary of each candidate span.
  • Final Scoring: LLaMA-3 acts as a "Span Scorer," comparing the span-level summaries against the queries to rank the best matches.
  • Refinement: Non-Maximum Suppression (NMS) is applied to prune overlapping or redundant candidates, resulting in the final predicted moment.
Figure 3: Detailed architecture of the Moment-GPT pipeline, showing the flow from raw video frames and queries to the final localized span.

Experimental Evaluation

Moment-GPT was rigorously tested on three benchmark datasets: QVHighlights (diverse YouTube content), Charades-STA (indoor activities), and ActivityNet-Captions (outdoor activities).

The results demonstrate that Moment-GPT sets a new state-of-the-art (SOTA) for zero-shot VMR. On the QVHighlights benchmark, it achieved a 58.3% [email protected], representing a 5.9% absolute improvement over previous zero-shot methods. Notably, this performance even surpassed several fully supervised models, such as VTimeLLM, highlighting the power of leveraging high-quality off-the-shelf LLMs. Similar gains were observed across Charades-STA (+4.8% [email protected]) and ActivityNet (+2.5% [email protected]).

Key Insights and Discussion

The success of Moment-GPT stems from several key design choices:

  • Semantic Abstraction: By converting video frames into captions, the model operates in a high-level semantic space, which naturally filters out low-level visual noise and redundancy.
  • LLMs as Reasoning Engines: Using LLaMA-3 for both query cleaning and similarity scoring allows the system to handle complex linguistic structures that traditional cross-modal encoders might miss.
  • Hierarchical Understanding: The division of labor—MiniGPT-v2 for fine-grained details and Video-ChatGPT for temporal summaries—mimics human-like video comprehension.

Despite its success, the pipeline faces challenges regarding inference latency due to the multiple calls required to large-scale models. Future work may focus on optimizing this multi-agent coordination or distilling the reasoning capabilities into more efficient, end-to-end architectures. Nonetheless, Moment-GPT proves that sophisticated VMR can be achieved without the need for expensive, task-specific training data.

Summary updated

BibTeX
@inproceedings{xu2025zero,
  author = {Yifang Xu and Yunzhuo Sun and Benxiang Zhai and Ming Li and Wenxin Liang and Yang Li and Sidan Du},
  title = {{Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.32971},
}
Figure