FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding

Fufangchen Zhao, Jinhu Fu, Jiachen Lei, Jiahong Wu, Xiangxiang Chu, Danfeng YanarXiv 2026 · arXiv

TL;DR

Current video AI models excel at multiple-choice questions about unusual or counterfactual events, but they often "cheat" by exploiting text clues in the question options. This paper introduces FADE, a two-stage training framework that forces models to actively discover and explain visual anomalies on their own as textual hints are progressively stripped away.

What Problem Does This Paper Address?

Counterfactual video understanding tests whether an AI truly grasps physical laws and common sense—for instance, recognizing the absurdity of a rocket suddenly surfacing from the water instead of a jet ski.

However, the authors point out a major flaw in how we currently evaluate these models: almost all benchmarks rely on Multiple-Choice Questions (MCQs). These questions and candidate options act as "textual anchors" that inadvertently leak the target event. Instead of autonomously discovering an anomaly, the model simply has to perform text-guided verification.

When you remove these textual options—moving from an MCQ to an open-ended question or a simple prompt to "describe the video"—even state-of-the-art models like GPT-5.6 experience massive performance drops. They fail because they don't actually know where or what the anomaly is without a text hint to guide their attention.

Figure 2: Anchor-fading profiles of representative Video-MLLMs on DualityVidQA-test (Both) under our evaluation protocol. Each panel shows one model; bars report absolute scores and lines report retention relative to MCQ. The consistent decline from MCQ to OQA and captioning reveals strong reliance on instance-specific textual anchors.

Methods

To bridge this gap, the authors present FADE (Fading Textual Anchors for Counterfactual Discovery and Explanation), an evidence-first, two-stage training paradigm:

  1. Stage I: Evidence-Internalized Supervised Fine-Tuning (SFT): This stage forces the model to ground its internal representations in actual visual evidence. By using a lightweight projector and a dual-branch re-grounding mechanism, the model learns to map its hidden response states directly to the specific video frames where the physical anomaly occurs. This ensures the model is actually "looking" at the right evidence.
  2. Stage II: Fading-Anchor Reinforcement Learning (RL): Once the model is visually grounded, it undergoes a "scaffold-and-consolidate" curriculum. The model is fed the same video across three progressively harder formats: MCQ $\rightarrow$ Open-ended Question Answering (OQA) $\rightarrow$ Captioning. Using Group Reward-Decoupled Normalization Policy Optimization (GDPO), the model is rewarded for maintaining correct answers even as the text anchors completely fade away.

Additionally, the authors designed an automated evaluation pipeline that reformulates existing MCQ datasets into aligned OQA and captioning tasks, removing the need for expensive manual data curation.

Datasets

The framework was trained and evaluated on two primary datasets:

  • DualityVidQA: The training split (containing over 104,000 factual and counterfactual video samples) was used for SFT and RL training. The test split was used to evaluate performance across all three text-fading levels.
  • IPV-Bench: A benchmark consisting exclusively of counterfactual videos, used to test the model's generalization capabilities under reduced textual guidance.

Key Insights and Ideas

  • The MCQ Illusion: High multiple-choice accuracy in video models is often an illusion sustained by text-based hints rather than true visual understanding.
  • Task-Agnostic Grounding: By forcing internal hidden states to align with visual anomaly timestamps during SFT, the model develops an intrinsic ability to spot weird events regardless of how it is prompted.
  • Curriculum Fading: Rather than training on unconstrained tasks from scratch, progressively "fading" the text anchors allows the model to transfer its text-guided reasoning capabilities into fully autonomous discovery.

Key Results

FADE (built on top of the open-source Qwen3-VL-8B baseline) demonstrates remarkable robustness against the removal of textual hints, outperforming proprietary models like GPT-5.6 on open-ended tasks.

  • High Performance Retention: On the DualityVidQA-test set, FADE retains 90.4% of its MCQ performance when transitioned to Open QA, and 67.4% when transitioned to pure captioning. For comparison, GPT-5.6 only retains 48.1% and 30.7% respectively.
  • Absolute Scores: On IPV-Bench captioning (where no question is provided), FADE achieves an accuracy score of 60.2%, vastly outperforming GPT-5.6's 25.5% and the baseline Qwen3-VL-8B's 16.1%.
Figure 1: Performance comparison between FADE and its base model as instance-specific textual anchors progressively fade. (a) Both models answer the anchor-rich MCQ correctly; after the options and then the question-specific cue are removed, only FADE consistently discovers and explains the counterfactual event. (b) Aggregate results on DualityVidQA-test and IPV-Bench. Bars report absolute scores, while lines report performance retained relative to MCQ.

Limitations and Future Work

While FADE marks a significant step toward unconstrained video understanding, it has a few limitations:

  • Dependence on Temporal Annotations: Stage I (SFT) requires precise timestamp annotations of when the counterfactual event occurs in the training data.
  • LLM-as-a-Judge Evaluation: Evaluating open-ended answers and captions requires a frozen semantic LLM verifier, which can introduce minor evaluation biases.
  • Future Directions: The authors suggest extending this fading-anchor strategy to other multimodal domains where models suffer from text-bias, such as complex visual document reasoning or long-form video summarization.

Summary updated

Abstract

Counterfactual video understanding evaluates whether models grasp physical and commonsense regularities. However, existing multiple-choice question (MCQ) benchmarks inadvertently leak target events through their questions and candidate options. This reduces the core challenge from active discovery to text-guided verification. In this paper, we present FADE, an effective training framework for counterfactual discovery and explanation. Our method is built on an evidence-first, two-stage training paradigm. First, evidence-internalized supervised fine-tuning grounds the model's predictions in decisive visual anomalies. Second, we apply a fading-anchor reinforcement learning strategy that progressively removes textual guidance, compelling the model to independently discover and explain evidence. To rigorously evaluate this capability, we also introduce an effective pipeline that converts existing MCQ datasets into aligned MCQ, open-ended question answering (OQA), and captioning tasks without requiring additional data curation. Our simple approach yields strong results. Using Qwen3-VL-8B as the baseline, FADE achieves state-of-the-art strict paired scores across all three tasks on DualityVidQA-test and IPV-Bench, outperforming GPT-5.6. In specific, when transitioning from constrained MCQs to unconstrained OQA and captioning, our model demonstrates remarkable robustness. Its performance retention is 90.4% and 67.4% on DualityVidQA-test-substantially higher than the 48.1% and 30.7% retained by GPT-5.6. We hope this simple framework can serve as a solid baseline for future research in unconstrained counterfactual video understanding.
BibTeX
@misc{zhao2026fade,
  author = {Fufangchen Zhao and Jinhu Fu and Jiachen Lei and Jiahong Wu and Xiangxiang Chu and Danfeng Yan},
  title = {{FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding}},
  year = {2026},
  eprint = {2608.10764},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.10764},
}
Figure