Cross Modality Bias in Visual Question Answering: A Causal View With Possible Worlds VQA

Ali Vosoughi, Shijian Deng, Songyang Zhang, Yapeng Tian, Chenliang Xu, Jiebo Luo

Introduction

Visual Question Answering (VQA) models often rely on "spurious correlations" rather than genuine multimodal reasoning. For instance, a model might consistently answer "green" when it sees the word "grass" in a question, regardless of the actual color depicted in the image. This paper introduces Possible Worlds VQA (PW-VQA), a novel causal framework designed to address these biases by modeling the subjective experience of annotators.

The Challenge of Cross-Modality Bias

Existing debiasing techniques typically target either language or vision bias in isolation. However, the authors argue that these biases are deeply intertwined. Reducing bias in one modality often inadvertently increases it in the other because both are confounded during data collection. Human annotators bring their own unique backgrounds and "possible worlds" into the process, influencing both how questions are phrased and how images are interpreted.

Methodology: Possible Worlds VQA (PW-VQA)

The PW-VQA framework employs causal inference to disentangle these biases.

The Annotator as a Confounder

A key innovation is treating the annotator's subjective experience as an unobserved confounder ($U$). This accounts for phenomena like the "Rubin’s Vase" effect, where the same visual stimulus can be perceived as either a vase or two faces depending on the observer's mental state.

The Possible Worlds VQA framework illustrates how different annotator perspectives (World 1 vs. World 2) lead to different interpretations of the same image, and how the model reasons across these possibilities.

Causal and Counterfactual Reasoning

The framework utilizes several advanced techniques:

  1. Anticausal Modeling: The ground-truth answer is modeled as a cause of the question and image, rather than just an effect.
  2. Explain-Away (EA) Fusion: A new fusion strategy that allows the model to prioritize the modality providing the most logical signal.
  3. Counterfactual Inference: During inference, the model compares its full multimodal prediction against a "biased" baseline (e.g., seeing only the question). By calculating the "Total Indirect Effect," the model can subtract the influence of known biases to arrive at a more reasoned answer.

Experimental Results

The researchers evaluated PW-VQA on major benchmarks, including VQA-CP v2 (designed for changing priors) and the standard VQA v2.

State-of-the-Art Performance

PW-VQA achieved state-of-the-art results on VQA-CP v2. Crucially, unlike many previous methods, it maintained robust performance on the standard VQA v2 benchmark, demonstrating that its debiasing does not come at the cost of general accuracy.

Breakthrough in Numerical Reasoning

The most significant improvement was observed in numerical questions. Using the SAN backbone, accuracy on counting-related questions soared from 12.4% to 57.7%. This suggests that the causal approach enables the model to actually learn the logic of counting rather than relying on statistical guesses.

Conclusion

PW-VQA represents a significant step forward in creating trustworthy VQA systems. By acknowledging the role of the human annotator as a source of confounded bias and employing counterfactual reasoning, the framework achieves superior performance and better generalization. Future work will focus on integrating external world knowledge to handle questions that require context beyond the immediate visual and linguistic input.

Summary updated

Abstract

To increase the generalization capability of VQA systems, many recent studies have tried to de-bias spurious language or vision associations that shortcut the question or image to the answer. Despite these efforts, the literature fails to address the confounding effect of vision and language simultaneously. As a result, when they reduce bias learned from one modality, they usually increase bias from the other. In this paper, we first model a confounding effect that causes language and vision bias simultaneously, then propose a counterfactual inference to remove the influence of this effect. The model trained in this strategy can concurrently and efficiently reduce vision and language bias. To the best of our knowledge, this is the first work to reduce biases resulting from confounding effects of vision and language in VQA, leveraging causal explain-away relations. We accompany our method with an explain-away strategy, pushing the accuracy of the questions with numerical answers results compared to existing methods that have been an open problem. The proposed method outperforms the state-of-the-art methods in VQA-CP v2 datasets.
BibTeX
@article{vosoughi2024cross,
  author = {Ali Vosoughi and Shijian Deng and Songyang Zhang and Yapeng Tian and Chenliang Xu and Jiebo Luo},
  title = {{Cross Modality Bias in Visual Question Answering: A Causal View With Possible Worlds VQA}},
  journal = {IEEE Trans. Multimedia},
  year = {2024},
  doi = {10.1109/tmm.2024.3380259},
}
Figure