TL;DR
V-STORM introduces a structure-aware framework that transforms videos into Spatio-Temporal Scene Graphs (STSGs) to generate a massive, verifiable dataset for video reasoning. By linking every step of a reasoning chain to specific timestamps and bounding boxes, it provides the granular supervision necessary for Multimodal Large Language Models (MLLMs) to perform faithful, evidence-grounded video understanding.
What Problem Does This Paper Address?
While current MLLMs are increasingly adept at identifying what happens in a video, they frequently stumble when asked to perform fine-grained reasoning that requires knowing exactly when and where supporting evidence appears. This often leads to "hallucinated" reasoning—where a model provides the correct answer but bases it on the wrong visual cues.
The authors identify a major bottleneck: the lack of scalable, high-quality data that provides step-level supervision. Most existing datasets either offer only video-level answers (lacking "how" the model got there) or rely on expensive, non-scalable human labeling for spatial/temporal grounding. Without verifiable reasoning chains, it is difficult to diagnose whether a model is truly reasoning or just making a lucky guess based on linguistic priors.
Methods
The core of the paper is the V-STORM framework, which moves away from direct end-to-end data generation in favor of a graph-grounded approach.
- Hybrid Perception & STSG Construction: The system first processes raw video using a "semantic proposer" (Florence-2) to identify objects and a "tracker" (SAM2) to maintain their identities across frames. These observations are aggregated into a hierarchical Spatio-Temporal Scene Graph (STSG), which organizes objects, their trajectories, and their dynamic relations into a unified structure.
- Programmatic Question Generation: Instead of asking an LLM to "invent" questions, V-STORM samples valid evidence paths directly from the STSG. These paths are then "rendered" into Question-Answer pairs accompanied by a Grounded Chain of Thought (GCoT).
- Traceable Verification: Because every step in the GCoT is derived from the graph, it is inherently linked to specific metadata (timestamps and coordinates). A verifier VLLM then cross-checks the generated text against the actual visual evidence to ensure factual consistency.
Datasets
The authors introduce two primary resources:
- V-STORM-260k: A large-scale training set containing 260,000 samples with GCoT annotations.
- V-STORM Benchmark: A high-quality, human-verified evaluation set of 3,300 samples across 11 task categories (e.g., frame-level grounding, temporal event reasoning, and complex combination tasks).
The dataset is built upon several public video sources, including SAV, ActivityNet, and InternVid, covering diverse real-world domains like sports, cooking, and traffic.
Key Insights and Ideas
- Localization vs. Composition Gap: A key finding is that current MLLMs are actually quite good at locating evidence (high grounding scores) but fail at composing those observations into multi-step reasoning. This "compositional bottleneck" is often hidden by standard answer-only evaluations.
- The Power of Structure: By using STSGs as an intermediate representation, the framework avoids the "black box" nature of LLM-based data generation. It ensures that every reasoning step is "traceable" back to a specific pixel region at a specific time.
- GCoT > CCoT: Fine-grained, grounded reasoning traces (GCoT) are significantly more effective for training than coarse-grained, language-only reasoning traces (CCoT). Explicitly forcing the model to cite spatial and temporal evidence during training leads to much higher faithfulness.
Key Results
The authors evaluated several state-of-the-art models and found that even leading commercial models struggle with the benchmark's complex reasoning tasks.
- Fine-tuning Impact: Fine-tuning a relatively small model (Qwen3-VL-2B) on V-STORM-260k led to dramatic improvements. Its overall accuracy jumped from 52.83% to 79.42%, outperforming much larger commercial models.
- Reasoning Gains: The most significant boost was seen in Compositional Reasoning, where accuracy soared from roughly 20% to over 73%.
- Grounding Faithfulness: The Grounding Score (measuring how well the model's cited evidence matches the truth) improved significantly, reaching the highest levels among all evaluated models.
Limitations and Future Work
- Perception Dependency: The quality of the STSG—and thus the entire dataset—is bounded by the performance of the initial perception models (Florence-2 and SAM2). Errors in tracking or object identification can propagate through the pipeline.
- Complexity Limits: While the graph-based approach is scalable, it may struggle to capture highly abstract or subjective "high-level" reasoning that isn't easily represented as a node or edge in a scene graph.
- Future Directions: The authors suggest that V-STORM could be extended to even longer-form videos and more diverse reasoning types, potentially serving as a foundation for training more reliable embodied AI agents.
Summary updated