Introduction
Video temporal grounding (VTG) is the task of identifying the start and end timestamps of an event described by a natural language query within an untrimmed video. While critical for video retrieval and editing, VTG remains challenging because user-provided queries are frequently underspecified or vague. For instance, a query like "Man starts surfing" lacks the visual context—such as the color of a surfboard or the specific motion—required for precise temporal alignment.
Existing approaches often struggle with this ambiguity. Specialist models trained on specific datasets lack zero-shot generalization, while general-purpose Multimodal Large Language Models (MLLMs) are proficient at reasoning but poor at outputting precise numerical timestamps. This paper introduces ED-VTG (Enrich and Detect), a framework that leverages the descriptive power of MLLMs to refine queries before performing localization with a dedicated perception module.
Methodology: The ED-VTG Framework
ED-VTG decomposes temporal grounding into a two-stage "cascaded" process: enrichment followed by detection.
1. Query Enrichment
The model, initialized with Video-LLaMA, first processes the video and the original query to generate an enriched query. By captioning the target action with visual details found in the video (e.g., "The man with a yellow surfboard slowly runs to start surfing"), the model creates a more discriminative textual representation that simplifies the subsequent grounding task.
2. Interval Detection
Rather than forcing the LLM to output raw timestamps—a task for which it is ill-suited—ED-VTG utilizes a specialized interval decoder. When the LLM generates an enriched query, it emits a special <INT> token. The hidden state of this token is fed into a lightweight decoder that predicts the center and width of the temporal interval. This separation allows the use of standard detection losses, such as L1 and generalized IoU (gIoU), to optimize for precision.
3. Training with Multiple Instance Learning (MIL)
To train the enrichment stage without perfect ground-truth captions, the authors employ a Multiple Instance Learning (MIL) objective. During training, the model generates predictions using both the original and enriched queries. It then backpropagates through the path that yields the lower grounding loss. This strategy allows the model to learn from noisy pseudo-labels generated by off-the-shelf captioners while autonomously deciding when enrichment is beneficial.
Experimental Results
ED-VTG was evaluated across diverse tasks, including Single-Query Temporal Grounding (STG) and Video Paragraph Grounding (VPG), using datasets like Charades-STA, ActivityNet-Captions, and TACoS.
Performance Highlights
- Zero-Shot Superiority: In zero-shot settings on Charades-STA, ED-VTG achieved a [email protected] of 59.5, significantly outperforming prior LLM-based models like HawkEye (50.6).
- Fine-Tuning Excellence: On the TACoS dataset, ED-VTG reached 32.4 mIoU, surpassing the previous best LLM-based model (TimeChat) by 14 absolute points.
- Qualitative Impact: Qualitative analysis demonstrates that enriched queries provide the necessary context for the model to distinguish between similar actions or to find the exact start of a subtle movement.
Discussion and Limitations
The success of ED-VTG highlights that captioning and grounding are complementary tasks. By decoupling language reasoning from spatial-temporal perception, the model achieves both flexibility and precision.
However, limitations remain. The current architecture relies solely on visual and textual inputs, omitting audio cues which are often vital for grounding (e.g., "a person laughs"). Additionally, the model can occasionally "over-enrich" queries by adding extraneous actions, leading to elongated interval predictions. Future work will focus on integrating audio modalities and scaling pre-training to even larger video-text corpora.
Summary updated