Evidence-Backed Video Question Answering

Shijie Wang, Honglu Zhou, Ziyang Wang, Ran Xu, Caiming Xiong, Silvio Savarese, Chen Sun, Juan Carlos NieblesarXiv 2026 · arXiv

TL;DR

This paper introduces Evidence-Backed Video Question Answering (E-VQA), a new task that requires Video LLMs to justify their textual answers with precise, pixel-level spatio-temporal evidence. To support this, the authors release ST-Evidence, a human-verified evaluation benchmark, and ST-Evidence-Instruct, a 160k-scale dataset that dramatically improves the visual grounding and reasoning capabilities of video models.

What Problem Does This Paper Address?

While modern Video Large Language Models (Video LLMs) excel at answering questions about video content, they largely operate as "black boxes." They provide textual answers without any verifiable proof that they actually perceived the correct visual cues, making it hard to distinguish genuine understanding from language priors or hallucinations.

Existing explainability methods rely on text rationales or sparse bounding boxes. However, these approaches fail to capture complex, fluid video dynamics like severe occlusions, continuous state changes, and non-rigid deformations (e.g., a wire bending or a liquid pouring). To establish true trust and traceability in high-stakes domains like autonomous driving or medical analysis, models must provide a dense, pixel-level "proof" of their reasoning.

Methods

The authors formalize the E-VQA task, where a model must output a triplet: (1) the semantic answer, (2) the supporting temporal segments, and (3) the supporting spatial evidence represented as dense, tracked segmentation masks (masklets).

The E-VQA framework requiring a semantic answer, temporal segments, and dense spatial masklets.

To generate training data at scale without prohibitive human annotation costs, they design a bidirectional, automated data construction pipeline:

  1. Grounding-to-Semantics (Leveraging Existing Masks): They take videos that already have dense object masks and captions (from the ViCaS dataset) and use frontier VLMs (Qwen3-VL and Gemini) to synthesize complex QA pairs grounded in those specific objects, followed by a rigorous multi-stage text validation filter.
  2. Semantics-to-Grounding (Leveraging Existing QA): For datasets with complex QA pairs but no visual annotations, they use Qwen3-VL to identify critical "evidence objects" and predict their bounding boxes at specific timestamps. They then use SAM-3 (Segment Anything 3) to propagate these sparse bounding boxes into dense, tracked spatio-temporal masks across the entire video.

Finally, they build a baseline model by fine-tuning UniPixel (3B and 7B variants) using LoRA on the vision encoder and LLM, while fully training the mask segmentation decoder.

Datasets

The paper introduces three distinct dataset splits for the E-VQA task:

  • ST-Evidence-Gen: A generative benchmark containing 2,706 questions and 2,548 videos, featuring human-verified temporal segments and dense spatial masks annotated at 6 FPS.
  • ST-Evidence-MCQ: A multiple-choice discriminative variant containing 1,298 questions and 348 videos, designed to evaluate general Video LLMs that lack native mask-generation heads.
  • ST-Evidence-Instruct: A massive instruction-tuning dataset containing 160k triplets across 30k videos, generated entirely via the automated pipeline to bridge high-level reasoning with pixel-level tracking.

Key Insights and Ideas

  • The Grounding-Accuracy Decoupling: High QA accuracy does not imply strong visual perception. Many models achieve ~80% accuracy on text answers but perform near random-guess levels (~25%) when asked to select the correct supporting spatial mask.
  • Scaling Compute Alone Fails: Increasing parameter scale within a model family consistently boosts textual QA accuracy, but improvements in temporal and spatial evidence grounding remain marginal.
  • Architecture Over Scale: Architectural innovations matter more than raw size for grounding. For example, the compact Qwen3-VL-4B matches or outperforms the massive Qwen2.5-VL-72B on grounding metrics, largely due to its interleaved text-timestamp alignment mechanism.
  • Proving "Evidential" Value: To prove that their annotated masks actually contain the answers, the authors pixelated/corrupted the evidence regions. This caused a massive drop in QA accuracy (~20%), while corrupting non-evidence regions caused only a minor drop, validating that the benchmark successfully targets causal visual cues.

Key Results

  • State-of-the-Art Evaluation: Proprietary models lead the benchmark, with Gemini-2.5-Pro achieving the highest QA accuracy (83.7%) and temporal grounding (49.9 t-mean). However, its spatial grounding tops out at a modest 44.0 $\mathcal{J}\&\mathcal{F}$ score, highlighting the difficulty of the task.
  • Instruction Tuning Gains: Fine-tuning the grounded baseline model on ST-Evidence-Instruct yields massive performance leaps over size-matched UniPixel baselines. The 7B model achieved an improvement of +27.2 t-mean on temporal grounding and +13.8 $\mathcal{J}\&\mathcal{F}$ on spatial grounding.
  • Qualitative Superiority: As shown in the qualitative analysis, baseline models often fail to track complete interactions or highlight irrelevant background objects. In contrast, the proposed fine-tuned model successfully segments critical entities across the exact causal time span.
Qualitative comparison showing the fine-tuned model successfully tracking causal entities compared to baselines.

Limitations and Future Work

  • Multi-Task Instruction Bottleneck: Current grounded LLMs struggle to handle long, multi-task instructions in a single inference pass. To evaluate them fairly, the authors had to use a sequential, multi-turn dialogue strategy. Future architectures need to support unified, single-pass generation for complex multi-task outputs.
  • Architectural Integration: The authors note that truly solving E-VQA requires moving away from loose proxy segmenters and instead developing fundamental advances in model architecture and integrated training objectives that natively bind linguistic logic to pixel-level tracking.

Summary updated

Abstract

Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.
BibTeX
@misc{wang2026evidence,
  author = {Shijie Wang and Honglu Zhou and Ziyang Wang and Ran Xu and Caiming Xiong and Silvio Savarese and Chen Sun and Juan Carlos Niebles},
  title = {{Evidence-Backed Video Question Answering}},
  year = {2026},
  eprint = {2607.11862},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.11862},
}
Figure