V-STORM: A Structure-Aware Grounded Chain of Thought Dataset for Faithful and Fine-Grained Spatio-Temporal Video Reasoning

Dian Jiao, Yuqian Yuan, Jiaqi Zhu, Mingjian Gao, Siwen Tan, Yunqi Cao, Jinhui Li, Sijing Li, Zhenxuan Fan, Haoyang Wang, Wenqiao Zhang, Siliang TangarXiv 2026 · arXiv

TL;DR

V-STORM introduces a structure-aware framework that transforms videos into Spatio-Temporal Scene Graphs (STSGs) to generate a massive, verifiable dataset for video reasoning. By linking every step of a reasoning chain to specific timestamps and bounding boxes, it provides the granular supervision necessary for Multimodal Large Language Models (MLLMs) to perform faithful, evidence-grounded video understanding.

What Problem Does This Paper Address?

While current MLLMs are increasingly adept at identifying what happens in a video, they frequently stumble when asked to perform fine-grained reasoning that requires knowing exactly when and where supporting evidence appears. This often leads to "hallucinated" reasoning—where a model provides the correct answer but bases it on the wrong visual cues.

The authors identify a major bottleneck: the lack of scalable, high-quality data that provides step-level supervision. Most existing datasets either offer only video-level answers (lacking "how" the model got there) or rely on expensive, non-scalable human labeling for spatial/temporal grounding. Without verifiable reasoning chains, it is difficult to diagnose whether a model is truly reasoning or just making a lucky guess based on linguistic priors.

Methods

The core of the paper is the V-STORM framework, which moves away from direct end-to-end data generation in favor of a graph-grounded approach.

  1. Hybrid Perception & STSG Construction: The system first processes raw video using a "semantic proposer" (Florence-2) to identify objects and a "tracker" (SAM2) to maintain their identities across frames. These observations are aggregated into a hierarchical Spatio-Temporal Scene Graph (STSG), which organizes objects, their trajectories, and their dynamic relations into a unified structure.
  2. Programmatic Question Generation: Instead of asking an LLM to "invent" questions, V-STORM samples valid evidence paths directly from the STSG. These paths are then "rendered" into Question-Answer pairs accompanied by a Grounded Chain of Thought (GCoT).
  3. Traceable Verification: Because every step in the GCoT is derived from the graph, it is inherently linked to specific metadata (timestamps and coordinates). A verifier VLLM then cross-checks the generated text against the actual visual evidence to ensure factual consistency.
Figure 2: V-STORM Data Construction Pipeline. The pipeline consists of three stages: (i) Hybrid Perception & STSG Construction, (ii) Programmatic Question Generation, and (iii) Traceable Verification. The STSG serves as the core asset that connects raw video to generated structured samples.

Datasets

The authors introduce two primary resources:

  • V-STORM-260k: A large-scale training set containing 260,000 samples with GCoT annotations.
  • V-STORM Benchmark: A high-quality, human-verified evaluation set of 3,300 samples across 11 task categories (e.g., frame-level grounding, temporal event reasoning, and complex combination tasks).

The dataset is built upon several public video sources, including SAV, ActivityNet, and InternVid, covering diverse real-world domains like sports, cooking, and traffic.

Figure 3: Distribution of video lengths in the Training Split and Benchmark, and video topics in the Benchmark.

Key Insights and Ideas

  • Localization vs. Composition Gap: A key finding is that current MLLMs are actually quite good at locating evidence (high grounding scores) but fail at composing those observations into multi-step reasoning. This "compositional bottleneck" is often hidden by standard answer-only evaluations.
  • The Power of Structure: By using STSGs as an intermediate representation, the framework avoids the "black box" nature of LLM-based data generation. It ensures that every reasoning step is "traceable" back to a specific pixel region at a specific time.
  • GCoT > CCoT: Fine-grained, grounded reasoning traces (GCoT) are significantly more effective for training than coarse-grained, language-only reasoning traces (CCoT). Explicitly forcing the model to cite spatial and temporal evidence during training leads to much higher faithfulness.

Key Results

The authors evaluated several state-of-the-art models and found that even leading commercial models struggle with the benchmark's complex reasoning tasks.

  • Fine-tuning Impact: Fine-tuning a relatively small model (Qwen3-VL-2B) on V-STORM-260k led to dramatic improvements. Its overall accuracy jumped from 52.83% to 79.42%, outperforming much larger commercial models.
  • Reasoning Gains: The most significant boost was seen in Compositional Reasoning, where accuracy soared from roughly 20% to over 73%.
  • Grounding Faithfulness: The Grounding Score (measuring how well the model's cited evidence matches the truth) improved significantly, reaching the highest levels among all evaluated models.
Figure 4: Definition of Answer Accuracy, Reasoning Score, and Grounding Score.

Limitations and Future Work

  • Perception Dependency: The quality of the STSG—and thus the entire dataset—is bounded by the performance of the initial perception models (Florence-2 and SAM2). Errors in tracking or object identification can propagate through the pipeline.
  • Complexity Limits: While the graph-based approach is scalable, it may struggle to capture highly abstract or subjective "high-level" reasoning that isn't easily represented as a node or edge in a scene graph.
  • Future Directions: The authors suggest that V-STORM could be extended to even longer-form videos and more diverse reasoning types, potentially serving as a foundation for training more reliable embodied AI agents.

Summary updated

Abstract

While MLLMs can often identify what happens in a video, they still struggle with fine-grained spatio-temporal reasoning, which requires precise grounding when and where the supporting evidence appears and composing such evidence into a coherent reasoning chain. This challenge is exacerbated by the lack of scalable data construction and evaluation pipelines that provide granular, verifiable supervision linking reasoning steps to explicit spatio-temporal evidence.To address this gap, we present V-STORM, a structure-aware framework that transforms videos into Spatio-Temporal Scene Graphs (STSGs) and derives Grounded Chain of Thought (GCoT) from validated evidence paths, making each reasoning step traceable to structured and verifiable spatio-temporal evidence.Under this framework, we construct a 260K-sample training set and a 3K-sample human-verified benchmark, both featuring GCoT annotations that link reasoning to explicit spatio-temporal evidence.Experiments reveal a clear separation between evidence localization and evidence composition: current MLLMs can often identify relevant spatio-temporal cues, yet still fail to integrate them into correct multi-step reasoning, exposing a bottleneck that answer-only evaluation largely obscures. Moreover, fine-tuning with V-STORM substantially improves both answer accuracy and grounding faithfulness, validating its effectiveness as supervision for faithful and evidence-grounded video reasoning. Our dataset, benchmark, and source code will be publicly released.
BibTeX
@misc{jiao2026v,
  author = {Dian Jiao and Yuqian Yuan and Jiaqi Zhu and Mingjian Gao and Siwen Tan and Yunqi Cao and Jinhui Li and Sijing Li and Zhenxuan Fan and Haoyang Wang and Wenqiao Zhang and Siliang Tang},
  title = {{V-STORM: A Structure-Aware Grounded Chain of Thought Dataset for Faithful and Fine-Grained Spatio-Temporal Video Reasoning}},
  year = {2026},
  eprint = {2611.09511},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09511},
}
Figure