Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, Lifu Huang

Abstract

Video Large Language Models (Video-LLMs) have demonstrated remarkable capabilities in coarse-grained video understanding; however, they struggle with fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM, a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner. We identify that current Video-LLMs have limitations for fine-grained video understanding since they lack effective temporal modeling and timestamp representation. In light of this, we sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge to represent timestamps. To optimize the training of Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with simple video-captioning tasks and progressively introducing video temporal grounding tasks of increasing complexity. To further enhance Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded VideoQA dataset by an automatic annotation pipeline. Extensive experiments demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA, but also shows great potential as a versatile video assistant for general video understanding.

The Challenge of Temporal Blindness

While current Video-LLMs are impressive at describing the general gist of a video (e.g., "a person is cooking"), they are notoriously limited in "fine-grained temporal grounding"—the ability to pinpoint exactly when an event starts and ends. This "temporal blindness" stems from two primary bottlenecks:

  1. Ineffective Temporal Modeling: Most models encode video frames independently using image encoders. While this captures spatial details, it ignores the temporal relationships and motion dynamics between frames.
  2. Numerical Inefficiency: LLMs are optimized for next-token prediction of discrete words, not for reasoning over continuous floating-point numbers. Representing timestamps as plain text (e.g., "12.5 seconds") is computationally inefficient and frequently leads to "hallucinated" or inaccurate timings.
Grounded-VideoLLM enables fine-grained temporal understanding through tasks like temporal referring, localization, and reasoning using discrete temporal tokens.

Architectural Innovation: Two-Stream Encoding and Temporal Tokens

Grounded-VideoLLM addresses these issues through a multi-pronged architectural approach designed to treat time as a first-class citizen within the LLM's vocabulary.

Two-Stream Encoding

The model processes video segments through two parallel paths. The Spatial Stream utilizes a standard image encoder on keyframes to capture detailed appearance. Simultaneously, the Temporal Stream employs a specialized video encoder (InternVideo2) on dense frames to extract motion representations. These dual representations are fused and fed into the LLM as "soft prompts," ensuring the model understands both what objects are present and how they are moving.

Unified Temporal Tokens

Instead of relying on text-based numbers, the authors extend the LLM’s vocabulary with $M$ discrete tokens (e.g., <0> to <300>) representing relative positions in a video. This allows the LLM to treat time as just another "word," sharing the same embedding space as text. By turning continuous time into discrete tokens, the LLM can leverage its internal attention mechanisms more effectively to "look" at relevant video segments when generating a timestamp.

The Grounded-VideoLLM architecture features a two-stream encoder (spatial and temporal) and an extended LLM vocabulary for discrete timestamp tokens.

Progressive Training and Data Curation

The training of Grounded-VideoLLM follows a "coarse-to-fine" learning curve across three stages:

  1. Video-Caption Alignment: Training projectors to connect visual features to the LLM.
  2. Temporal Token Alignment: Specifically training the model to use the new temporal tokens for grounding tasks.
  3. Multi-Task Instruction Tuning: Fine-tuning on a diverse mix of grounding, reasoning, and conversation tasks.

To further enhance temporal reasoning, the researchers curated a new Grounded VideoQA dataset containing 17,000 samples using an automatic GPT-4 pipeline. This dataset requires the model to not only answer a question but also provide the specific timestamps that support its reasoning.

Key Results and Performance

Grounded-VideoLLM (based on Phi-3.5) achieved significant performance gains across major benchmarks:

  • Temporal Sentence Grounding: On the Charades-STA benchmark, it achieved a 36.8 mIoU, surpassing the previous state-of-the-art (HawkEye) by +3.1 points.
  • Grounded VideoQA: On NExT-GQA, it reached an Acc@GQA of 26.7%, a +2.4% improvement over existing models.
  • General Understanding: Despite its focus on grounding, it remained a top performer on general benchmarks like VCG-Bench (3.24 avg score) and MVBench (60.0 avg score), proving that sharpening temporal precision does not compromise general intelligence.

Conclusion

Grounded-VideoLLM represents a significant step forward in video understanding by successfully integrating motion dynamics and discrete temporal representations. By treating time as a linguistic token and employing a two-stream encoding strategy, the model overcomes the traditional limitations of Video-LLMs, setting new standards for fine-grained temporal grounding and reasoning.

Summary updated

Abstract

Video Large Language Models (Video-LLMs) have demonstrated remarkable capabilities in coarse-grained video understanding, however, they struggle with fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM, a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner. We identify that current Video-LLMs have limitations for fine-grained video understanding since they lack effective temporal modeling and timestamp representation. In light of this, we sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge to represent timestamps. To optimize the training of Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with simple video-captioning tasks and progressively introducing video temporal grounding tasks of increasing complexity. To further enhance Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded VideoQA dataset by an automatic annotation pipeline. Extensive experiments demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA, but also shows great potential as a versatile video assistant for general video understanding.
BibTeX
@misc{wang2024grounded,
  author = {Haibo Wang and Zhiyang Xu and Yu Cheng and Shizhe Diao and Yufan Zhou and Yixin Cao and Qifan Wang and Weifeng Ge and Lifu Huang},
  title = {{Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models}},
  year = {2024},
  eprint = {2410.03290},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.03290},
}
Figure