TL;DR
This paper introduces Evidence-Backed Video Question Answering (E-VQA), a new task that requires Video LLMs to justify their textual answers with precise, pixel-level spatio-temporal evidence. To support this, the authors release ST-Evidence, a human-verified evaluation benchmark, and ST-Evidence-Instruct, a 160k-scale dataset that dramatically improves the visual grounding and reasoning capabilities of video models.
What Problem Does This Paper Address?
While modern Video Large Language Models (Video LLMs) excel at answering questions about video content, they largely operate as "black boxes." They provide textual answers without any verifiable proof that they actually perceived the correct visual cues, making it hard to distinguish genuine understanding from language priors or hallucinations.
Existing explainability methods rely on text rationales or sparse bounding boxes. However, these approaches fail to capture complex, fluid video dynamics like severe occlusions, continuous state changes, and non-rigid deformations (e.g., a wire bending or a liquid pouring). To establish true trust and traceability in high-stakes domains like autonomous driving or medical analysis, models must provide a dense, pixel-level "proof" of their reasoning.
Methods
The authors formalize the E-VQA task, where a model must output a triplet: (1) the semantic answer, (2) the supporting temporal segments, and (3) the supporting spatial evidence represented as dense, tracked segmentation masks (masklets).
To generate training data at scale without prohibitive human annotation costs, they design a bidirectional, automated data construction pipeline:
- Grounding-to-Semantics (Leveraging Existing Masks): They take videos that already have dense object masks and captions (from the ViCaS dataset) and use frontier VLMs (Qwen3-VL and Gemini) to synthesize complex QA pairs grounded in those specific objects, followed by a rigorous multi-stage text validation filter.
- Semantics-to-Grounding (Leveraging Existing QA): For datasets with complex QA pairs but no visual annotations, they use Qwen3-VL to identify critical "evidence objects" and predict their bounding boxes at specific timestamps. They then use SAM-3 (Segment Anything 3) to propagate these sparse bounding boxes into dense, tracked spatio-temporal masks across the entire video.
Finally, they build a baseline model by fine-tuning UniPixel (3B and 7B variants) using LoRA on the vision encoder and LLM, while fully training the mask segmentation decoder.
Datasets
The paper introduces three distinct dataset splits for the E-VQA task:
- ST-Evidence-Gen: A generative benchmark containing 2,706 questions and 2,548 videos, featuring human-verified temporal segments and dense spatial masks annotated at 6 FPS.
- ST-Evidence-MCQ: A multiple-choice discriminative variant containing 1,298 questions and 348 videos, designed to evaluate general Video LLMs that lack native mask-generation heads.
- ST-Evidence-Instruct: A massive instruction-tuning dataset containing 160k triplets across 30k videos, generated entirely via the automated pipeline to bridge high-level reasoning with pixel-level tracking.
Key Insights and Ideas
- The Grounding-Accuracy Decoupling: High QA accuracy does not imply strong visual perception. Many models achieve ~80% accuracy on text answers but perform near random-guess levels (~25%) when asked to select the correct supporting spatial mask.
- Scaling Compute Alone Fails: Increasing parameter scale within a model family consistently boosts textual QA accuracy, but improvements in temporal and spatial evidence grounding remain marginal.
- Architecture Over Scale: Architectural innovations matter more than raw size for grounding. For example, the compact Qwen3-VL-4B matches or outperforms the massive Qwen2.5-VL-72B on grounding metrics, largely due to its interleaved text-timestamp alignment mechanism.
- Proving "Evidential" Value: To prove that their annotated masks actually contain the answers, the authors pixelated/corrupted the evidence regions. This caused a massive drop in QA accuracy (~20%), while corrupting non-evidence regions caused only a minor drop, validating that the benchmark successfully targets causal visual cues.
Key Results
- State-of-the-Art Evaluation: Proprietary models lead the benchmark, with Gemini-2.5-Pro achieving the highest QA accuracy (83.7%) and temporal grounding (49.9 t-mean). However, its spatial grounding tops out at a modest 44.0 $\mathcal{J}\&\mathcal{F}$ score, highlighting the difficulty of the task.
- Instruction Tuning Gains: Fine-tuning the grounded baseline model on ST-Evidence-Instruct yields massive performance leaps over size-matched UniPixel baselines. The 7B model achieved an improvement of +27.2 t-mean on temporal grounding and +13.8 $\mathcal{J}\&\mathcal{F}$ on spatial grounding.
- Qualitative Superiority: As shown in the qualitative analysis, baseline models often fail to track complete interactions or highlight irrelevant background objects. In contrast, the proposed fine-tuned model successfully segments critical entities across the exact causal time span.
Limitations and Future Work
- Multi-Task Instruction Bottleneck: Current grounded LLMs struggle to handle long, multi-task instructions in a single inference pass. To evaluate them fairly, the authors had to use a sequential, multi-turn dialogue strategy. Future architectures need to support unified, single-pass generation for complex multi-task outputs.
- Architectural Integration: The authors note that truly solving E-VQA requires moving away from loose proxy segmenters and instead developing fundamental advances in model architecture and integrated training objectives that natively bind linguistic logic to pixel-level tracking.
Summary updated