Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, Lifu HuangSubmission to ICLR 2025 · Withdrawn

TL;DR

Grounded-VideoLLM tackles the challenge of fine-grained temporal grounding in video large language models (Video-LLMs). By combining a dual-stream architecture (separating spatial appearance from dense temporal motion) with discrete relative temporal tokens ($<0>$ to $<300>$), the model avoids clumsy text-based floating-point timestamps and learns precise temporal boundaries. Across temporal sentence grounding, dense captioning, and grounded VideoQA, it sets new state-of-the-art results while maintaining top-tier performance on general video benchmarks.

What Problem Does This Paper Address?

While modern Video-LLMs excel at answering broad questions or generating coarse captions for short clips, they struggle to pinpoint when events happen with fine granularity. The authors identify two primary bottlenecks:

  1. Weak Temporal Modeling: Most Video-LLMs encode sparsely sampled frames independently using static image encoders (e.g., CLIP ViT) and rely on the language model's positional embeddings to infer time, ignoring subtle motion dynamics and frame-to-frame relationships.
  2. Inefficient Timestamp Representation: Many models represent timestamps as plain floating-point text (e.g., "from 102.3 to 120.1 seconds"). Autoregressive language models inherently struggle with continuous numerical data and digit-level tokenization.

Methods

Grounded-VideoLLM builds on off-the-shelf multimodal architectures (evaluating both Phi-3.5-Vision-Instruct (3.8B) and LLaVA-1.5 / Vicuna-1.5 (7B)) using three key innovations:

  • Two-Stream Video Encoding: Videos ($T=96$ frames) are split into $K=12$ segments:
  • Spatial Stream: A keyframe per segment is passed through an image encoder at high resolution ($336 \times 336$) with mild $2\text{D}$ pooling ($\sigma_S = 2$), preserving spatial detail ($144$ tokens/frame).
  • Temporal Stream: Dense frames ($8$ frames/segment) are encoded at lower spatial resolution ($224 \times 224$) via a pretrained video expert (InternVideo2-1B) with aggressive spatial pooling ($\sigma_T = 4$) to retain full temporal resolution ($16$ tokens/frame).
  • Segment features from both streams are projected via MLPs and concatenated, totaling $3{,}264$ visual tokens.
  • Unified Relative Temporal Tokens: Rather than generating raw numbers, the LLM vocabulary is expanded with $M+1$ special tokens ($\langle 0 \rangle$ to $\langle M \rangle$, with $M=300$) representing uniform relative anchor points along the video duration. Timestamps and textual responses are decoded seamlessly in a single autoregressive sequence.
  • Three-Stage Progressive Training:
  • Video-Caption Alignment: Aligns visual projectors on coarse video-caption pairs (encoders and LLM frozen).
  • Temporal Token Alignment: Introduces temporal tokens, unfreezing projectors, token embeddings, and the output classification head to map temporal tokens directly into the LLM’s semantic space and video timeline.
  • Multi-Task Instruction Tuning: Tunes projectors, temporal embeddings, and LLM LoRA parameters across mixed grounding and general video QA/instruction tasks.
+-------------------------------------------------------------+
|                     Input Video (T=96)                      |
+------------------------------+------------------------------+
                               |
               Divide into K=12 segments
                               |
         +---------------------+---------------------+
         |                                           |
   Spatial Stream                             Temporal Stream
(Keyframe @ 336x336)                        (8 Frames @ 224x224)
         |                                           |
   Image Encoder                               Video Encoder
    (CLIP ViT)                                (InternVideo2)
         |                                           |
     2D Pooling                                  2D Pooling
  (144 tokens/seg)                            (128 tokens/seg)
         |                                           |
       S-MLP                                       T-MLP
         +---------------------+---------------------+
                               |
              Concatenate Tokens (3,264 visual tokens)
                               |
                               v
                     +-------------------+
                     |   Base LLM        | <--- Text Instruction
                     +-------------------+
                               |
            Unified Autoregressive Generation
               e.g., "From <250> to <280>, he bites..."

Datasets

The authors leverage a structured suite of datasets across training and evaluation:

  • Pre-training & Instruction Tuning:
  • Stage 1: ~1.28M video-text pairs from WebVid-10M, Panda-70M, and InternVid-10M.
  • Stage 2: ~336K samples from VTimeLLM-Stage2, Moment-10M, and InternVid-G.
  • Stage 3: Grounded dialogue, sentence grounding, and dense captioning datasets (RTL, DiDeMo, HiREST, QuerYD, COIN, ViTT, YouCook2, VTG-IT), alongside general instruction datasets (VideoChat2, ShareGPT4Video, Kinetics, etc.).
  • Newly Curated Dataset (Grounded VideoQA): An automated dataset of 17K samples constructed from QVHighlights using GPT-4. Given segment descriptions and timestamps, GPT-4 generated causal/temporal QA pairs; negative options were selected by sampling distractors based on sentence embedding cosine similarities ($0.2$ to $0.9$).
  • Evaluation Benchmarks: Charades-STA, ActivityNet-Grounding, ActivityNet-Captions, NExT-GQA, Open-Ended VideoQA (MSVD, MSRVTT, ActivityNet-QA), and general Video-LLM benchmarks (VCG-Bench, MVBench).

Key Insights and Ideas

  • Complementary Dual Streams over Pure Dense Vision: Ablations show that purely dense frames without the dual-stream setup compromise spatial recognition, while sparse frames degrade temporal localization. Combining a spatial keyframe encoder with a spatially downsampled dense video encoder achieves the optimal trade-off.
  • The Necessity of Token Alignment: Without Stage 2 (token alignment), attention weights for temporal tokens diffuse across irrelevant parts of the video. With it, attention weights sharp-focus on exact temporal boundaries, and PCA visualizations show that temporal token embeddings organize into a smooth, continuous chronological manifold.
  • Relative Tokens Beat Plain Text: Using 300 discrete relative tokens consistently outperformed raw text timestamps, particularly on longer videos where numerical formatting errors and precision drift are most severe.

Key Results

  • Temporal Sentence Grounding: On Charades-STA, Grounded-VideoLLM (Phi3.5) achieved 36.8 mIoU and 19.7 [email protected], beating previous state-of-the-art end-to-end models like HawkEye (33.7 mIoU, 14.5 [email protected]). On ActivityNet-Grounding, it achieved 36.1 mIoU.
  • Dense Video Captioning: Achieved a top SODA_c score of 6.0 and METEOR of 6.8 on ActivityNet-Captions, highlighting strong multi-event storyline comprehension.
  • Grounded VideoQA: Reached 26.7 Acc@GQA on NExT-GQA (+2.4 over prior methods), which climbs to 29.4 when including task-specific training data.
  • General Video Understanding: Scored 60.0 Avg on MVBench (outperforming VideoGPT+ at 58.7) and 3.24 Avg on VCG-Bench, notably achieving the highest Temporal Understanding (TU) score of 3.12 (+7% over comparable models).

Limitations and Future Work

  • Quantization Errors: Discretizing continuous time into $M=300$ uniform bins introduces minor rounding errors, which can impact sub-second action boundaries in very long videos.
  • Computational Footprint: Operating dual encoders across dense frames produces 3,264 visual tokens per clip, demanding substantial GPU memory and compute during both training and inference relative to sparse single-stream models.

Summary updated

Abstract

Video Large Language Models (Video-LLMs) have demonstrated remarkable capabilities in coarse-grained video understanding, however, they struggle with fine-grained temporal grounding. In this paper, we introduce $\textit{Grounded-VideoLLM}$, a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner. We identify that current Video-LLMs have limitations for fine-grained video understanding since they lack effective temporal modeling and timestamp representation. In light of this, we sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge to represent timestamps. To optimize the training of $\textit{Grounded-VideoLLM}$, we employ a multi-stage training scheme, beginning with simple video-captioning tasks and progressively introducing video temporal grounding tasks of increasing complexity. To further enhance $\textit{Grounded-VideoLLM}$'s temporal reasoning capability, we also curate a grounded VideoQA dataset by an automatic annotation pipeline. Extensive experiments demonstrate that $\textit{Grounded-VideoLLM}$ not only excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA, but also shows great potential as a versatile video assistant for general video understanding.
BibTeX
@misc{wang2024grounded,
  author = {Haibo Wang and Zhiyang Xu and Yu Cheng and Shizhe Diao and Yufan Zhou and Yixin Cao and Qifan Wang and Weifeng Ge and Lifu Huang},
  title = {{Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models}},
  year = {2024},
  eprint = {2410.03290},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.03290},
}

Loading related content...

Figure