Beyond Isolated Facts: Synthesizing Narrative and Grounded Supervision for VideoQA

Jianxin Liang, Tan Yue, Yuxuan Wang, Yueqian Wang, Zhihan Yin, Huishuai Zhang, Dongyan ZhaoSubmission to ICLR 2026 · Rejected

TL;DR

Current Video Question Answering (VideoQA) models are bottlenecked by training on fragmented "bags-of-facts"—isolated question-answer pairs that miss causal and temporal structure. This paper introduces a data-centric paradigm that transforms raw QA pairs into dense, structured supervisory signals: Question-Based Paraphrasing (QBP) to synthesize holistic narrative paragraphs, and Question-Based Captioning (QBC) to generate fine-grained visual rationales. Training Multimodal LLMs exclusively on this synthetic data sets new state-of-the-art results across benchmarks, improves cross-dataset generalization, and accelerates training convergence by over 2.5×.

What Problem Does This Paper Address?

Standard VideoQA datasets present video content as a disjointed set of atomic question-answer pairs (e.g., "What is the weather?", "Why is the person resting?", "How does the person react?"). While individually accurate, treating these pairs as independent targets strips away the semantic, temporal, and causal connective tissue of the scene.

Because models are forced to learn from this "bag-of-facts," they frequently rely on shallow linguistic correlations and hallucinate when forced to perform complex multi-step reasoning. Previous data-synthesis efforts attempted to fix data scarcity by generating more isolated QA pairs or generic captions, failing to fix this fundamental structural deficit.

Methods

Rather than training on original QA annotations, the authors propose replacing raw QA pairs entirely with two synergistic, synthesized supervision formats:

Raw QA Group  ──► [LLM: GPT-4o / DeepSeek]  ──► Holistic Narrative (QBP)
(video-level)                                           │
                                                        ▼
Raw QA Pair   ──► [MLLM: Gemini]            ──► Visual Rationale (QBC) ──► Unified SFT Training
(instance-level + video)                                                  (Next-token prediction)
  1. Question-Based Paraphrasing (QBP) — Global Narrative Context: An LLM synthesizes all the diverse inquiries (what, how, why) associated with a single video into a cohesive, logic-infused paragraph. By using the ground-truth QA pairs as factual anchors, QBP reconstructs the overarching narrative and event dependencies without inventing new information.

  2. Question-Based Captioning (QBC) — Local Visual Grounding: Conditioned on the video, a specific question, and its ground-truth answer, a Multimodal Large Language Model (MLLM) is prompted to produce a targeted visual rationale. Crucially, the model is forbidden from simply restating the answer; it must articulate the exact visual evidence justifying why that answer is correct.

Downstream VideoQA backbones are fine-tuned via standard autoregressive next-token prediction solely on the union of these two synthetic corpora ($\mathcal{D}_{\text{train}} = \mathcal{D}^{\text{QBP}} \cup \mathcal{D}^{\text{QBC}}$).

Datasets

The authors synthesize data using seed annotations from three prominent VideoQA and video-language benchmarks:

  • NExT-QA: Dense causal and temporal QA (~3.8k videos, ~34k QA pairs).
  • STAR: Situated reasoning in real-world videos (~3k videos, ~45k QA pairs).
  • DiDeMo: Spatially/temporally localized video descriptions (~2k videos, ~7k QA pairs; used primarily as a sparse seed source).

Models are evaluated on NExT-QA and STAR in both in-domain and zero-shot cross-dataset transfer settings.

Key Insights and Ideas

  • Inquiries as Event Graphs: Human-annotated questions for a video are not random; their collective intent probes static entities, dynamic processes, and causality. Re-assembling them into unified text recovers the underlying event structure.
  • Semantic Density Speeds Up Training: QBP condenses ~30k NExT-QA pairs into ~3k narrative paragraphs. Despite seeing 10× fewer instances, models trained on QBP reach peak performance in ~220 steps compared to ~600 steps for raw QA—a >2.5× convergence speedup. One structured paragraph provides denser gradient signals than multiple fragmented queries.
  • Mitigating Dataset Overfitting: Models fine-tuned on raw NExT-QA gain +1.9% in-domain but degrade on unseen STAR data (-1.1%). In contrast, QBP supervision boosts in-domain accuracy while simultaneously increasing cross-dataset performance (+2.2%), showing that narrative learning abstracts away dataset-specific phrasing artifacts.

Key Results

  • State-of-the-Art Accuracy:
  • Applying QBP+QBC to Qwen2.5-VL-7B establishes a new SOTA of 80.8% on NExT-QA (+4.6% over the base model) and 73.3% on STAR (+2.7%).
  • On Qwen2.5-VL-3B, it pushes STAR accuracy from 67.5% to 72.5% (+5.0%) and NExT-QA to 76.8% (+2.5%).
  • Consistent gains were also observed on MiMo-VL-SFT 7B (STAR: 52.0% $\rightarrow$ 56.2%).
  • QBC Scaling: Scaling QBC synthetic rationales from 3.5k up to 29k consistently improves in-domain accuracy and drives transfer accuracy on STAR from 66.5% to nearly 70.0% (+3.5%).
  • Data Quality: Blind human evaluation of 200 samples on a 1–5 scale showed high factual consistency (4.21 for QBP, 4.35 for QBC) and strong visual grounding (4.38 for QBC).

Limitations and Future Work

  • "Justified Fabrication" in QBC: When ground-truth visual cues are ambiguous or subtle, MLLMs given the correct answer occasionally invent plausible-sounding visual descriptions to rationalize the answer.
  • Temporal Incoherence in QBP: If seed QA annotations lack clear chronological timestamps, the LLM occasionally misorders sequential actions.
  • Redundancy: Videos with repetitive questions can lead to repetitive concluding sentences in synthesized narratives.
  • Future Directions: Incorporating explicit temporal interval modeling and automated verification filters to catch rationalization hallucinations during synthesis.

Summary updated

Abstract

The performance of Video Question Answering (VideoQA) models is fundamentally constrained by the nature of their supervision, which typically consists of isolated, factual question-answer pairs. This "bag-of-facts" approach fails to capture the underlying narrative and causal structure of events, limiting models to a shallow understanding of video content. To move beyond this paradigm, we introduce a framework to synthesize richer supervisory signals. We propose two complementary strategies: Question-Based Paraphrasing (QBP), which synthesizes the diverse inquiries (what, how, why) from a video's existing set of question-answer pairs into a holistic narrative paragraph that reconstructs the video's event structure; and Question-Based Captioning (QBC), which generates fine-grained visual rationales, grounding the answer to each question in specific, relevant evidence. Leveraging powerful generative models, we use this synthetic data to train VideoQA models under a unified next-token prediction objective. Extensive experiments on STAR and NExT-QA validate our approach, demonstrating significant accuracy gains and establishing new state-of-the-art results, such as improving a 3B model to 72.5\% on STAR (+4.9\%) and a 7B model to 80.8\% on NExT-QA. Beyond accuracy, our analysis reveals that both QBP and QBC substantially enhance cross-dataset generalization, with QBP additionally accelerating model convergence by over 2.5x. These results demonstrate that shifting data synthesis from isolated facts to narrative coherence and grounded rationales yields a more accurate, efficient, and generalizable training paradigm.
BibTeX
@misc{liang2025beyond,
  author = {Jianxin Liang and Tan Yue and Yuxuan Wang and Yueqian Wang and Zhihan Yin and Huishuai Zhang and Dongyan Zhao},
  title = {{Beyond Isolated Facts: Synthesizing Narrative and Grounded Supervision for VideoQA}},
  year = {2025},
  eprint = {2509.24445},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24445},
}
Figure