Enrich and Detect: Video Temporal Grounding with Multimodal LLMs

Shraman Pramanick, Effrosyni Mavroudi, Yale Song, Rama Chellappa, Lorenzo Torresani, Triantafyllos Afouras

Introduction

Video temporal grounding (VTG) is the task of identifying the start and end timestamps of an event described by a natural language query within an untrimmed video. While critical for video retrieval and editing, VTG remains challenging because user-provided queries are frequently underspecified or vague. For instance, a query like "Man starts surfing" lacks the visual context—such as the color of a surfboard or the specific motion—required for precise temporal alignment.

Existing approaches often struggle with this ambiguity. Specialist models trained on specific datasets lack zero-shot generalization, while general-purpose Multimodal Large Language Models (MLLMs) are proficient at reasoning but poor at outputting precise numerical timestamps. This paper introduces ED-VTG (Enrich and Detect), a framework that leverages the descriptive power of MLLMs to refine queries before performing localization with a dedicated perception module.

Methodology: The ED-VTG Framework

ED-VTG decomposes temporal grounding into a two-stage "cascaded" process: enrichment followed by detection.

1. Query Enrichment

The model, initialized with Video-LLaMA, first processes the video and the original query to generate an enriched query. By captioning the target action with visual details found in the video (e.g., "The man with a yellow surfboard slowly runs to start surfing"), the model creates a more discriminative textual representation that simplifies the subsequent grounding task.

2. Interval Detection

Rather than forcing the LLM to output raw timestamps—a task for which it is ill-suited—ED-VTG utilizes a specialized interval decoder. When the LLM generates an enriched query, it emits a special <INT> token. The hidden state of this token is fed into a lightweight decoder that predicts the center and width of the temporal interval. This separation allows the use of standard detection losses, such as L1 and generalized IoU (gIoU), to optimize for precision.

The ED-VTG architecture combines an LLM for query enrichment with a specialized interval decoder for precise temporal localization. The right panel illustrates the Multiple Instance Learning (MIL) training objective.

3. Training with Multiple Instance Learning (MIL)

To train the enrichment stage without perfect ground-truth captions, the authors employ a Multiple Instance Learning (MIL) objective. During training, the model generates predictions using both the original and enriched queries. It then backpropagates through the path that yields the lower grounding loss. This strategy allows the model to learn from noisy pseudo-labels generated by off-the-shelf captioners while autonomously deciding when enrichment is beneficial.

Experimental Results

ED-VTG was evaluated across diverse tasks, including Single-Query Temporal Grounding (STG) and Video Paragraph Grounding (VPG), using datasets like Charades-STA, ActivityNet-Captions, and TACoS.

Performance Highlights

  • Zero-Shot Superiority: In zero-shot settings on Charades-STA, ED-VTG achieved a [email protected] of 59.5, significantly outperforming prior LLM-based models like HawkEye (50.6).
  • Fine-Tuning Excellence: On the TACoS dataset, ED-VTG reached 32.4 mIoU, surpassing the previous best LLM-based model (TimeChat) by 14 absolute points.
  • Qualitative Impact: Qualitative analysis demonstrates that enriched queries provide the necessary context for the model to distinguish between similar actions or to find the exact start of a subtle movement.
Qualitative comparison showing how ED-VTG enriches vague user queries with visual details to improve grounding accuracy compared to ground truth.

Discussion and Limitations

The success of ED-VTG highlights that captioning and grounding are complementary tasks. By decoupling language reasoning from spatial-temporal perception, the model achieves both flexibility and precision.

However, limitations remain. The current architecture relies solely on visual and textual inputs, omitting audio cues which are often vital for grounding (e.g., "a person laughs"). Additionally, the model can occasionally "over-enrich" queries by adding extraneous actions, leading to elongated interval predictions. Future work will focus on integrating audio modalities and scaling pre-training to even larger video-text corpora.

Summary updated

BibTeX
@inproceedings{pramanick2025enrich,
  author = {Shraman Pramanick and Effrosyni Mavroudi and Yale Song and Rama Chellappa and Lorenzo Torresani and Triantafyllos Afouras},
  title = {{Enrich and Detect: Video Temporal Grounding with Multimodal LLMs}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure