Diagnosing Causal Reasoning in Vision-Language Models via Structured Relevance Graphs

Dhita Putri Pratama, Soyeon Caren Han, Yihao Ding

TL;DR

Large Vision-Language Models (LVLMs) often get the right answer for the wrong reasons by relying on surface-level patterns rather than logic. This paper introduces Vision-Language Causal Graphs (VLCGs) and the ViLCaR benchmark to "look under the hood," revealing that providing models with explicit causal structures significantly improves their reasoning consistency even when final answer accuracy remains the same.

What Problem Does This Paper Address?

Current LVLMs (like GPT-4o or Qwen-VL) perform impressively on Visual Question Answering (VQA) benchmarks. However, researchers have long suspected that these models are "cheating" by using spurious correlations—for example, seeing a white dress and a cake and guessing "wedding" without actually understanding the causal relationship between the entities.

The core issue is that existing benchmarks primarily measure Question Answering (QA) accuracy. If a model gets the answer right, we assume it reasoned correctly. This paper argues that this is a flawed assumption. There is a missing diagnostic step: relevance identification. Before a model can reason, it must correctly identify which specific objects, attributes, and social assumptions are causally relevant to the query. This paper seeks to disentangle "getting the right answer" from "having a valid reason."

Methods

The authors propose a new framework centered on two main contributions:

  1. Vision-Language Causal Graphs (VLCGs): Unlike standard scene graphs that attempt to map every object in an image, a VLCG is query-conditioned. It is a directed graph that encodes only the minimal set of objects, attributes, and relations necessary to answer a specific question. Crucially, it includes scene-grounded assumptions—the "common sense" or cultural knowledge (e.g., "a person in a suit at a counter is likely a server") required to bridge the gap between visual perception and a logical conclusion.
  2. The ViLCaR Benchmark: A diagnostic pipeline that breaks reasoning into three tasks:
  3. Causal Attribution (CA): Can the model identify the correct relevant attributes?
  4. Causal Inference (CI): Can the model generate a logical reasoning chain connecting those attributes to an answer?
  5. Question Answering (QA): Is the final prediction correct?

To evaluate these, they developed a graph-aligned metric that uses semantic similarity (BERT and Word2Vec) to check if the model's textual reasoning actually matches the "gold standard" causal graph.

Datasets

The authors introduced the ViLCaR dataset, containing 12.5K instances.

  • It was constructed by filtering existing datasets like VQA v2 and VCR (Visual Commonsense Reasoning) to remove "easy" questions that could be solved by simple object detection or spatial lookups.
  • They kept only questions requiring "Ladder of Causation" reasoning (associative, interventional, or counterfactual).
  • The causal graphs were initially generated using LVLMs and then rigorously pruned and verified by humans to ensure they represented the "minimal sufficient" causal path.

Key Insights and Ideas

  • Relevance is the Bottleneck: The study found that many model failures aren't due to a lack of "thinking power," but rather a failure to identify what information is relevant. When the authors "injected" the causal graph into the prompt, the models' reasoning improved significantly.
  • The Decoupling Effect: A major "aha moment" was observing that QA accuracy is decoupled from reasoning quality. Models often maintained the same accuracy score while their Causal Attribution (CA) scores fluctuated. This proves that high accuracy on current leaderboards is a poor proxy for actual intelligence or reliability.
  • Assumptions as Causal Links: The inclusion of "assumptions" in the VLCGs highlights that visual reasoning isn't just about pixels; it's about the invisible social and physical rules we apply to those pixels.

Key Results

The researchers benchmarked Qwen2.5-VL-7B across three settings: Zero-shot, Standard In-Context Learning (ICL), and VLCG-Augmented prompting.

  • Reasoning Consistency: Injecting VLCGs improved Causal Attribution by 6.6% and Causal Inference by 5.8% over the zero-shot baseline.
  • Accuracy vs. Logic: While reasoning scores (CA and CI) went up significantly with structural guidance, VQA Accuracy stayed almost flat (moving from 0.763 to 0.768).
  • Qualitative Shift: In one example, a zero-shot model failed to identify the roles of "server" and "customer" in a cafe, leading to an "I don't know" answer. When provided with the VLCG, the model correctly identified the roles and the underlying assumption about service, leading to a correct and logical answer.

Limitations and Future Work

  • Semantic Matching: The evaluation relies on approximate semantic alignment between the model's text and the graph. While scalable, it may not be as precise as a human judge.
  • Graph Construction: Creating these "gold standard" graphs is complex. While the authors used a semi-automated pipeline, scaling this to millions of images remains a challenge.
  • Future Directions: The authors suggest that future work should focus on using these graphs to "train" models to be more causally aware, rather than just using the graphs as prompts at inference time.

Summary updated

Abstract

Large Vision-Language Models (LVLMs) achieve strong performance on visual question answering benchmarks, yet often rely on spurious correlations rather than genuine causal reasoning. Existing evaluations primarily assess the correctness of the answers, making it unclear whether failures arise from limited reasoning capability or from misidentifying causally relevant information. We introduce Vision-Language Causal Graphs (VLCGs), a structured, query-conditioned representation that explicitly encodes causally relevant objects, attributes, relations, and scene-grounded assumptions. Building on this representation, we present ViLCaR, a diagnostic benchmark comprising tasks for Causal Attribution, Causal Inference, and Question Answering, along with graph-aligned evaluation metrics that assess relevance identification beyond final answer accuracy. Experiments in state-of-the-art LVLMs show that injecting structured relevance information significantly improves attribution and inference consistency compared to zero-shot and standard in-context learning. These findings suggest that current limitations in LVLM causal reasoning stem primarily from insufficient structural guidance rather than a lack of reasoning capacity.
BibTeX
@misc{pratama2026diagnosing,
  author = {Dhita Putri Pratama and Soyeon Caren Han and Yihao Ding},
  title = {{Diagnosing Causal Reasoning in Vision-Language Models via Structured Relevance Graphs}},
  year = {2026},
  eprint = {2602.20878},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.20878},
}
Figure