DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning

Tianrun Xu, Haoda Jing, Ye Li, Yuquan Wei, Jun Feng, Guanyu Chen, Haichuan Gao, Tianren Zhang, Feng Chen

Abstract

Multimodal Large Language Models (MLLMs) often exhibit "hallucinations" by relying on implicit language priors rather than visual evidence. We present DeFacto, a framework designed to enforce evidence-answer consistency through counterfactual training. By masking critical image regions and requiring models to abstain when evidence is missing, DeFacto ensures that models are "right for the right reasons." Our approach significantly improves accuracy and grounding across diverse visual reasoning benchmarks, including a massive +13.4% gain in spatial reasoning.

The Challenge: Beyond Spurious Correctness

While modern MLLMs are adept at "thinking with images," they frequently bypass genuine visual reasoning in favor of educated guesses based on text prompts. The authors identify three primary failure modes:

  1. Mislocalized Failure: The model attends to irrelevant regions and produces an incorrect answer.
  2. Spurious Correctness: The model attends to the wrong regions but happens to guess the correct answer—the "right for the wrong reasons" problem.
  3. Faithful Incorrectness: The model identifies the correct evidence but fails to interpret it, leading to an error.

The core challenge is enforcing evidence–answer consistency, ensuring that every correct answer is strictly supported by the appropriate visual evidence.

The DeFacto Framework

DeFacto treats visual reasoning as a verifiable process using three complementary training paradigms:

  • Positive Supervision: Models are trained on original images to predict both "evidence regions" (bounding boxes) and the correct answer.
  • Counterfactual Abstention: This is the framework's core innovation. By masking essential visual evidence, the model is penalized for guessing and rewarded for outputting "Unknown." This breaks the reliance on language shortcuts.
  • Random Masking: Irrelevant parts of the image are masked to ensure the model maintains its ability to find answers when the necessary evidence remains visible.
Figure 2: An example of DeFacto's reasoning process, showing the model's internal 'think' block and its ability to identify specific evidence regions before answering.

To optimize these objectives, the authors utilize Group Relative Policy Optimization (GRPO). They employ a composite reward function that scores answer correctness, format consistency (using <bbox> and <think> tags), and Region Selection Coherence—the overlap between the model's chosen boxes and actual evidence.

Datasets and Evaluation

The research introduces two significant datasets to support this training and evaluation:

  1. DeFacto-100K: A training dataset of 100,000 images. It uses a teacher model (Qwen2.5-VL) and an open-vocabulary detector (DINO-X) to automatically localize and mask evidence regions.
  2. DeFacto-1.5K: A high-quality, human-annotated benchmark consisting of 1,500 samples across 15 tasks, where humans have verified the exact bounding boxes required for each answer.

Key Results and Insights

Built on the Qwen2.5-VL-7B backbone, DeFacto achieved substantial performance gains:

  • General VQA: Accuracy rose from 67.0% to 73.8% on VQAv2 and from 53.2% to 63.1% on GQA.
  • Spatial Reasoning: On the VSR (Visual Spatial Reasoning) benchmark, DeFacto saw a massive +13.4% improvement.
  • Faithfulness: On the DeFacto-1.5K benchmark, the model's mean Average Precision (mAP) for evidence grounding skyrocketed from 0.7 to 29.3, indicating a much tighter link between visual perception and textual output.
Figure 6: Visualization of precise evidence grounding, where the model successfully localizes the specific subject required to answer a visual query.

Teaching a model when it cannot answer is as vital as teaching it how to answer. By rewarding "Unknown" in counterfactual scenarios, the model learns that the answer must derive from the image, not the prompt. Furthermore, jointly optimizing for "where to look" and "what to say" leads to a more robust model that generalizes better to complex, fine-grained tasks.

Limitations and Future Work

Despite its success, DeFacto saw a slight performance dip on text-heavy tasks like DocVQA (-5.2%), suggesting that current masking strategies may be too aggressive for dense document environments. Future work will explore more granular visual interactions, such as dynamic zooming or multi-step cropping, to further enhance fine-grained perception.

Summary updated

BibTeX
@inproceedings{xu2026defacto,
  author = {Tianrun Xu and Haoda Jing and Ye Li and Yuquan Wei and Jun Feng and Guanyu Chen and Haichuan Gao and Tianren Zhang and Feng Chen},
  title = {{DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=TcICHju9Mo},
}
Figure