Abstract
Video Moment Localization (VML) aims to identify specific segments in untrimmed videos based on natural language queries. However, existing models often fail to generalize across different video domains due to the "domain gap" and the structural mismatch between VML queries and the training data of pre-trained vision-language models (PVLs). We propose Moment-Guided Query Prompting (MGQP), a framework that decomposes complex queries into semantic primitives—Protagonist, Action, and Object—and refines them using visual cues. This approach enables better alignment with PVLs and enhances cross-domain transferability. Experimental results across four major benchmarks demonstrate that MGQP significantly outperforms existing baselines in zero-shot and cross-domain scenarios.
Introduction
The primary challenge in Video Moment Localization (VML) is the significant performance drop when models are tested on domains different from their training data. For instance, a model trained on cooking videos (TACoS) often fails to generalize to daily indoor activities (Charades-STA). This domain sensitivity arises because models overfit to specific visual contexts. Furthermore, while large-scale pre-trained vision-language models (PVLs) like CLIP offer robust general features, they are typically trained on short image captions. VML queries are often long and complex, creating a "query gap" that hinders the direct application of PVLs.
To address these issues, we introduce MGQP, which acts as a bridge between complex VML queries and the simplified semantic space of PVLs. By breaking down queries into fundamental components that are invariant across domains, MGQP facilitates more robust localization.
Methodology
The MGQP framework consists of three core modules designed to make natural language queries more "digestible" for frozen PVLs:
1. Prompt Primitive Constructor
Rather than processing a long sentence as a single unit, this module decomposes the query into three semantic "primitives": Protagonist (who), Action (doing what), and Object (to what). These primitives are then wrapped in templates (e.g., "The actions in this clip are [Action Chunk]") to align the query structure with the descriptive captions used during PVL pre-training.
2. Multimodal Prompt Refiner
To ensure the model focuses on relevant semantic cues, this module uses visual features from the video to dynamically weight the importance of each prompt primitive. If a specific object is visually prominent, the corresponding "Object Primitive" receives higher attention, ensuring the text prompts are "visually aware."
3. Holistic Prompt Incorporator
While primitives capture key semantic elements, they may miss broader context like temporal flow or causal links. The Holistic Prompt Incorporator uses a Transformer-based architecture with learnable global prompts to capture this "big picture" context, which is then fused with the local primitive prompts.
Experiments and Results
We evaluated MGQP on four benchmarks: TACoS (cooking), Charades-STA (indoor activities), DiDeMo (open domain), and YouCookII (instructional videos). The focus was on cross-domain transfer, training on one dataset and testing on another.
Key Results
- TACoS → Charades: MGQP achieved a Rank@1 (IoU=0.3) of 39.36%, significantly outperforming the MMN baseline (36.21%) and CoCoOp (30.75%).
- DiDeMo → YouCookII: The model reached 8.51% Rank@1 (IoU=0.3), a substantial improvement over the 5.31% achieved by 2D-TAN.
- Backbone Versatility: MGQP proved effective across various PVL backbones, including CLIP, ActionCLIP, and VideoCLIP, demonstrating its general-purpose utility as a transferability booster.
Conclusion
MGQP addresses the domain gap in VML by bridging the semantic divide between complex queries and pre-trained vision-language models. By decomposing queries into fundamental primitives and incorporating visual guidance, the framework achieves superior cross-domain generalization. While the current model excels at identifying actors and objects, future work will focus on modeling more complex temporal and causal relationships to handle highly intricate natural language queries.
Summary updated