Counterfactual Debiasing for Physical Audiovisual Commonsense Reasoning

Daoming Zong, Chaoyue Ding, Kaitao Chen, Yinsheng Li, Shuaiyu WangAAAI 2025 · DOI

TL;DR

This paper introduces a novel framework, CF-PACR, to combat visual bias in physical audiovisual commonsense reasoning (PACR) models. By leveraging counterfactual reasoning, CF-PACR effectively disentangles spurious visual correlations from genuine multimodal cues, leading to significantly improved accuracy and generalization on challenging datasets.

What Problem Does This Paper Address?

Humans possess an intuitive understanding of the physical world, a crucial aspect for developing safe and reliable AI systems. This "physical commonsense" involves understanding object properties, interactions, and affordances. While this reasoning is inherently multisensory, existing models for Physical Audiovisual Commonsense Reasoning (PACR) heavily rely on visual information. This reliance creates a "spurious correlation" problem: models learn to associate certain visual patterns with correct answers, even if these associations are coincidental and not based on true causal relationships. This leads to poor performance on "long-tail" scenarios, where object combinations or presentations are rare in the training data. For example, a model might incorrectly identify a ceramic object as wood if it has seen many "foam + wood" combinations during training, even if the visual cues for ceramic are present. This paper aims to mitigate this visual bias to improve the robustness and generalization of PACR models.

Methods

The core contribution is the Counterfactual Physical Audiovisual Commonsense Reasoning (CF-PACR) framework. It's designed to be model-agnostic and works by separating the "direct effects" of visual information from the "indirect effects" that leverage multimodal fusion.

Here's a breakdown:

  • Factual Reasoning: A standard PACR model is trained using both audio and visual information. This captures the "total effect" of all modalities.
  • Counterfactual Reasoning: A hypothetical scenario is created where only visual information is considered. This is achieved by blocking the direct pathway from visual features to predictions. The model is asked: "What would the prediction be if only visual information was available?" This isolates the "direct effect" of visual cues, including spurious correlations.
  • Debiasing: The framework calculates the "Total Effect" (TE) from factual reasoning and the "Natural Direct Effect" (NDE) from counterfactual reasoning. By subtracting the NDE from the TE, the spurious visual biases are removed, leaving the "Total Indirect Effect" (TIE), which represents the reliable multimodal reasoning.
  • Causal Graph: The method is grounded in a causal graph that explicitly models the relationships between question (Q), visual information (V), audio information (A), multimodal representation (M), and model prediction (Y).
  • Loss Function: The training incorporates a cross-entropy loss for both factual and counterfactual predictions, along with a Kullback-Leibler divergence term to minimize the difference between counterfactual predictions from the visual-only model and the audiovisual model.

Datasets

The primary dataset used is PACS (Physical Audiovisual Commonsense Reasoning). This dataset is designed for evaluating models on physical commonsense reasoning using both audio and visual modalities. It includes two tasks:

  • PACS-QA: Binary question answering about physical properties.
  • PACS-Material: Object material classification.

The paper notes that the PACS dataset exhibits an imbalanced distribution of object combinations, which is the source of the visual bias they aim to address.

Key Insights and Ideas

  • Visual bias is a significant bottleneck: The paper strongly demonstrates that over-reliance on visual cues, especially in imbalanced datasets, leads to spurious correlations and hinders generalization.
  • Counterfactual reasoning is key to disentangling effects: By simulating a "what if" scenario (e.g., "what if only visual input was available?"), the framework can isolate and remove the detrimental influence of visual shortcuts.
  • Direct vs. Indirect Effects: The distinction between direct visual effects (potentially biased) and indirect multimodal effects (more reliable) is crucial for understanding and mitigating bias.
  • Model-agnostic applicability: The CF-PACR framework can be integrated with various existing PACR models, offering a general solution rather than a model-specific one.

Key Results

  • Significant Accuracy Improvements: CF-PACR consistently improved the performance of baseline models. On average, it achieved an increase of approximately 2.82 percentage points in PACS-QA and 2.85 percentage points in PACS-Material.
  • Enhanced Generalization: The framework demonstrated superior performance on rare or underrepresented visual object combinations, which are typically challenging for models prone to visual bias.
  • Fusion Function Impact: The choice of fusion function for combining factual and counterfactual outcomes matters, with the Log-sigmoid Sum showing a preference.
  • Parameter Sensitivity: The parameter $\tau$, which controls the extent of debiasing, was found to be sensitive, with an optimal value (e.g., 0.6 for Log-sigmoid Sum) balancing bias mitigation and cue preservation.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while CF-PACR significantly reduces visual bias, some visually dominant object combinations might see a slight decline in accuracy, possibly due to the removal of "excessive" visual cues.
  • The effectiveness of the framework is demonstrated on the PACS dataset; its performance on other datasets with different types or degrees of visual bias is not explored.
  • Future Work:
    • The authors suggest that CF-PACR can be seamlessly applied to existing multimodal PACS models, implying potential for further integration and exploration with a wider range of models.
  • The framework's ability to generalize to other domains beyond physical commonsense reasoning could be investigated.

Summary updated

BibTeX
@inproceedings{zong2025counterfactual,
  author = {Daoming Zong and Chaoyue Ding and Kaitao Chen and Yinsheng Li and Shuaiyu Wang},
  title = {{Counterfactual Debiasing for Physical Audiovisual Commonsense Reasoning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i14.33675},
}
Figure