Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA

Ali Vosoughi, Shijian Deng, Songyang Zhang, Yapeng Tian, Chenliang Xu, Jiebo LuoarXiv 2023 · DOI · OpenReview

TL;DR

This paper introduces "Possible Worlds VQA" (PW-VQA), a novel causal framework to simultaneously address confounding biases from both vision and language in Visual Question Answering (VQA) systems. By employing a counterfactual "explain-away" strategy, PW-VQA significantly improves VQA performance, particularly on challenging numerical questions, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Visual Question Answering (VQA) systems aim to answer questions about images. However, many VQA models learn spurious correlations, or "biases," between visual features and answers, or between question phrasing and answers, rather than truly understanding the image content. Previous attempts to de-bias these systems often focused on either vision or language bias, but not both simultaneously. This led to a trade-off: reducing bias in one modality often increased it in the other. The paper highlights that this confounding effect, where vision and language biases are intertwined, is a critical challenge that needs to be addressed for VQA systems to generalize better. They also point out that human perception itself can be influenced by experience and memory (illustrated by the Rubin's vase example), which can translate into biases during dataset annotation.

Methods

The core of the proposed method, PW-VQA, is a causal framework that models VQA as a system where the answer influences the question and image, and then uses this understanding to de-bias the system. Key components include:

  • Causal Graph Formulation: They propose a causal graph that views the answer as a cause of the question and image, and then uses this to model the relationship between visual and linguistic information.
  • Explain-Away Network (Collider Bias): A novel "explain-away" network, modeled as a collider (Q → Â ← V), is introduced to simultaneously capture and mitigate both vision and language biases. This network helps distinguish between "good" and "bad" biases, aiming to retain useful information while removing spurious correlations.
  • Counterfactual Inference: A counterfactual approach is employed to remove the influence of confounding biases. This involves creating hypothetical scenarios (counterfactual worlds) to isolate and eliminate unwanted influences.
  • Experience as Unobserved Confounder: The paper also models the annotator's experience as an unobserved confounder, acknowledging that personal preferences can influence question-answer pairs in datasets.
  • Explain-Away Fusion Strategy (EA): A specific fusion strategy is proposed to combine multimodal information, leveraging the causal framework to produce a debiased answer.

Datasets

The paper primarily uses the VQA-CP v2 dataset for training and evaluation. This dataset is known for its "changing priors" characteristic, meaning the dataset is designed to expose VQA models that rely on spurious correlations. They also mention using the VQA v2 dataset for comparison.

Key Insights and Ideas

  • Simultaneous De-biasing is Crucial: The paper strongly argues that addressing vision and language biases concurrently is essential, as treating them in isolation leads to a detrimental trade-off.
  • Causality as a Solution: Framing VQA through a causal lens provides a powerful mechanism to disentangle true relationships from spurious correlations.
  • The "Explain-Away" Concept: The novel "explain-away" network, inspired by collider bias, is a key innovation for simultaneously modeling and mitigating cross-modal biases.
  • Numerical Questions are a Bottleneck: The paper highlights that numerical questions have been a persistent challenge for VQA systems, and their method shows significant improvements in this area.
  • Annotator Experience Matters: Recognizing the influence of annotator experience as a source of bias is an important step towards more robust VQA datasets and models.

Key Results

  • State-of-the-Art Performance: PW-VQA achieves state-of-the-art results on the VQA-CP v2 dataset.
  • Doubled Accuracy on Numerical Questions: The proposed method significantly improves accuracy on numerical questions, a previously open problem, by doubling the accuracy in some cases. For instance, with the SAN backbone, numerical accuracy increased from 12.4% to 37.6% with EA fusion and further to 57.7% with PW-VQA.
  • Generalizability: The improvements are consistent across different VQA backbones (SAN, S-MRL, UpDn), demonstrating the generalizability of the PW-VQA framework.
  • Reduced Bias: Qualitative examples show that PW-VQA is less biased by language or vision cues compared to regular VQA and CF-VQA. For example, in one instance, it correctly answers "No" to "Is she eating?" when a cat is present, unlike CF-VQA which confidently but incorrectly answers "cat."

Limitations and Future Work

  • Background Knowledge Dependency: The paper acknowledges that PW-VQA, like other VQA systems, still struggles with questions that require significant external background knowledge (e.g., "What year was this picture taken?" or "What bridge is this?").
  • Future Work: The authors suggest that using larger, pre-trained language-vision models (like generative BLIP decoder and CLIP encoders) can lead to further improvements. They also mention that the parameter $\alpha$ in their fusion strategy can be further tuned.

Summary updated

Abstract

To increase the generalization capability of VQA systems, many recent studies have tried to de-bias spurious language or vision associations that shortcut the question or image to the answer. Despite these efforts, the literature fails to address the confounding effect of vision and language simultaneously. As a result, when they reduce bias learned from one modality, they usually increase bias from the other. In this paper, we first model a confounding effect that causes language and vision bias simultaneously, then propose a counterfactual inference to remove the influence of this effect. The model trained in this strategy can concurrently and efficiently reduce vision and language bias. To the best of our knowledge, this is the first work to reduce biases resulting from confounding effects of vision and language in VQA, leveraging causal explain-away relations. We accompany our method with an explain-away strategy, pushing the accuracy of the questions with numerical answers results compared to existing methods that have been an open problem. The proposed method outperforms the state-of-the-art methods in VQA-CP v2 datasets.
BibTeX
@misc{vosoughi2023unveiling,
  author = {Ali Vosoughi and Shijian Deng and Songyang Zhang and Yapeng Tian and Chenliang Xu and Jiebo Luo},
  title = {{Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA}},
  year = {2023},
  eprint = {2305.19664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.19664},
}
Figure