TL;DR
Grounded-VideoLLM tackles the challenge of fine-grained temporal grounding in video large language models (Video-LLMs). By combining a dual-stream architecture (separating spatial appearance from dense temporal motion) with discrete relative temporal tokens ($<0>$ to $<300>$), the model avoids clumsy text-based floating-point timestamps and learns precise temporal boundaries. Across temporal sentence grounding, dense captioning, and grounded VideoQA, it sets new state-of-the-art results while maintaining top-tier performance on general video benchmarks.
What Problem Does This Paper Address?
While modern Video-LLMs excel at answering broad questions or generating coarse captions for short clips, they struggle to pinpoint when events happen with fine granularity. The authors identify two primary bottlenecks:
- Weak Temporal Modeling: Most Video-LLMs encode sparsely sampled frames independently using static image encoders (e.g., CLIP ViT) and rely on the language model's positional embeddings to infer time, ignoring subtle motion dynamics and frame-to-frame relationships.
- Inefficient Timestamp Representation: Many models represent timestamps as plain floating-point text (e.g.,
"from 102.3 to 120.1 seconds"). Autoregressive language models inherently struggle with continuous numerical data and digit-level tokenization.
Methods
Grounded-VideoLLM builds on off-the-shelf multimodal architectures (evaluating both Phi-3.5-Vision-Instruct (3.8B) and LLaVA-1.5 / Vicuna-1.5 (7B)) using three key innovations:
- Two-Stream Video Encoding: Videos ($T=96$ frames) are split into $K=12$ segments:
- Spatial Stream: A keyframe per segment is passed through an image encoder at high resolution ($336 \times 336$) with mild $2\text{D}$ pooling ($\sigma_S = 2$), preserving spatial detail ($144$ tokens/frame).
- Temporal Stream: Dense frames ($8$ frames/segment) are encoded at lower spatial resolution ($224 \times 224$) via a pretrained video expert (InternVideo2-1B) with aggressive spatial pooling ($\sigma_T = 4$) to retain full temporal resolution ($16$ tokens/frame).
- Segment features from both streams are projected via MLPs and concatenated, totaling $3{,}264$ visual tokens.
- Unified Relative Temporal Tokens: Rather than generating raw numbers, the LLM vocabulary is expanded with $M+1$ special tokens ($\langle 0 \rangle$ to $\langle M \rangle$, with $M=300$) representing uniform relative anchor points along the video duration. Timestamps and textual responses are decoded seamlessly in a single autoregressive sequence.
- Three-Stage Progressive Training:
- Video-Caption Alignment: Aligns visual projectors on coarse video-caption pairs (encoders and LLM frozen).
- Temporal Token Alignment: Introduces temporal tokens, unfreezing projectors, token embeddings, and the output classification head to map temporal tokens directly into the LLM’s semantic space and video timeline.
- Multi-Task Instruction Tuning: Tunes projectors, temporal embeddings, and LLM LoRA parameters across mixed grounding and general video QA/instruction tasks.
+-------------------------------------------------------------+
| Input Video (T=96) |
+------------------------------+------------------------------+
|
Divide into K=12 segments
|
+---------------------+---------------------+
| |
Spatial Stream Temporal Stream
(Keyframe @ 336x336) (8 Frames @ 224x224)
| |
Image Encoder Video Encoder
(CLIP ViT) (InternVideo2)
| |
2D Pooling 2D Pooling
(144 tokens/seg) (128 tokens/seg)
| |
S-MLP T-MLP
+---------------------+---------------------+
|
Concatenate Tokens (3,264 visual tokens)
|
v
+-------------------+
| Base LLM | <--- Text Instruction
+-------------------+
|
Unified Autoregressive Generation
e.g., "From <250> to <280>, he bites..."
Datasets
The authors leverage a structured suite of datasets across training and evaluation:
- Pre-training & Instruction Tuning:
- Stage 1: ~1.28M video-text pairs from WebVid-10M, Panda-70M, and InternVid-10M.
- Stage 2: ~336K samples from VTimeLLM-Stage2, Moment-10M, and InternVid-G.
- Stage 3: Grounded dialogue, sentence grounding, and dense captioning datasets (RTL, DiDeMo, HiREST, QuerYD, COIN, ViTT, YouCook2, VTG-IT), alongside general instruction datasets (VideoChat2, ShareGPT4Video, Kinetics, etc.).
- Newly Curated Dataset (Grounded VideoQA): An automated dataset of 17K samples constructed from QVHighlights using GPT-4. Given segment descriptions and timestamps, GPT-4 generated causal/temporal QA pairs; negative options were selected by sampling distractors based on sentence embedding cosine similarities ($0.2$ to $0.9$).
- Evaluation Benchmarks: Charades-STA, ActivityNet-Grounding, ActivityNet-Captions, NExT-GQA, Open-Ended VideoQA (MSVD, MSRVTT, ActivityNet-QA), and general Video-LLM benchmarks (VCG-Bench, MVBench).
Key Insights and Ideas
- Complementary Dual Streams over Pure Dense Vision: Ablations show that purely dense frames without the dual-stream setup compromise spatial recognition, while sparse frames degrade temporal localization. Combining a spatial keyframe encoder with a spatially downsampled dense video encoder achieves the optimal trade-off.
- The Necessity of Token Alignment: Without Stage 2 (token alignment), attention weights for temporal tokens diffuse across irrelevant parts of the video. With it, attention weights sharp-focus on exact temporal boundaries, and PCA visualizations show that temporal token embeddings organize into a smooth, continuous chronological manifold.
- Relative Tokens Beat Plain Text: Using 300 discrete relative tokens consistently outperformed raw text timestamps, particularly on longer videos where numerical formatting errors and precision drift are most severe.
Key Results
- Temporal Sentence Grounding: On Charades-STA, Grounded-VideoLLM (Phi3.5) achieved 36.8 mIoU and 19.7 [email protected], beating previous state-of-the-art end-to-end models like HawkEye (33.7 mIoU, 14.5 [email protected]). On ActivityNet-Grounding, it achieved 36.1 mIoU.
- Dense Video Captioning: Achieved a top SODA_c score of 6.0 and METEOR of 6.8 on ActivityNet-Captions, highlighting strong multi-event storyline comprehension.
- Grounded VideoQA: Reached 26.7 Acc@GQA on NExT-GQA (+2.4 over prior methods), which climbs to 29.4 when including task-specific training data.
- General Video Understanding: Scored 60.0 Avg on MVBench (outperforming VideoGPT+ at 58.7) and 3.24 Avg on VCG-Bench, notably achieving the highest Temporal Understanding (TU) score of 3.12 (+7% over comparable models).
Limitations and Future Work
- Quantization Errors: Discretizing continuous time into $M=300$ uniform bins introduces minor rounding errors, which can impact sub-second action boundaries in very long videos.
- Computational Footprint: Operating dual encoders across dense frames produces 3,264 visual tokens per clip, demanding substantial GPU memory and compute during both training and inference relative to sparse single-stream models.
Summary updated