Mitigating Language Bias of LMMs in Social Intelligence Understanding with Virtual Counterfactual Calibration

Peng Chen, Xiao-Yu Guo, Yuan-Fang Li, Xiaowang Zhang, Zhiyong FengEMNLP 2024 · DOI

Abstract

Large Multimodal Models (LMMs) often struggle with social intelligence tasks—understanding human intentions, emotions, and interactions—because they rely heavily on text-based "common sense" priors rather than visual evidence. This paper introduces DCVC (Distribution Calibration network with Virtual Counterfactual data augmentation), a framework that uses causal modeling and virtual "what-if" scenarios to calibrate model outputs. By addressing the "back-door" path of language bias, DCVC significantly boosts performance on social reasoning benchmarks like Social-IQ-2.0 without requiring expensive manual data labeling.

Introduction: The Language Bias Problem

Social intelligence is a critical frontier for AI. While LMMs like Video-LLaVA are powerful, they often exhibit a significant language bias. Because these models are pre-trained on vast text corpora, they develop strong internal priors. When presented with a video and a question (e.g., "How does not knowing the route make the woman feel?"), the model might ignore the visual context and select the most statistically likely text answer based on the question alone, leading to incorrect reasoning.

Figure 1: An example of language bias where the model predicts "Confused" based on text priors, ignoring the visual evidence of the woman laughing.

This "spurious correlation" between internal text knowledge and provided options often results in a "stochastic parrot" effect in social settings, where the model's prediction with video is nearly identical to its prediction without it.

Methodology: The DCVC Framework

The authors propose the DCVC framework to mitigate these biases through three core components:

  1. Structural Causal Model (SCM): Causal theory is used to identify how bias enters the system, specifically identifying internal knowledge as a "back-door" path that bypasses visual/audio context.
  2. Output Distribution Calibration (DC): A lightweight calibration network is added to re-weight the LMM's raw predictions against language priors, ensuring the final answer is grounded in multimodal context.
  3. Virtual Counterfactual Data Augmentation (VCDA): To train the calibration network without expensive video editing, the authors create "virtual" samples. By applying Gumbel noise to output distributions, they simulate how the model would react if visual context were different, allowing it to learn to ignore its biases.
Figure 3: The DCVC framework overview, illustrating the calibration network training and the generation of virtual counterfactual data.

Experimental Evaluation

The framework was evaluated on three major benchmarks: Social-IQ-2.0 (social intelligence), DeSIQ-2.0 (debiased social intelligence), and NExT-QA (general video QA).

Key Results

  • Performance Gains: On Social-IQ-2.0, adding DCVC to Video-LLaVA increased its score from 61.06% to 78.32%—a 17-point improvement.
  • Superiority over Fine-tuning: On DeSIQ-2.0, Video-LLaVA + DCVC (97.04%) outperformed models specifically fine-tuned on that dataset.
  • Generalizability: The performance boost on NExT-QA suggests that language bias is a universal hurdle for multimodal models beyond just social intelligence.

Conclusion

DCVC demonstrates that calibrating model outputs is often more effective and efficient than massive retraining. By performing counterfactual reasoning in the "probability space" rather than the "pixel space," the framework preserves beneficial linguistic priors while filtering out harmful social stereotypes. Future work will explore scaling the method to larger models and investigating the root causes of bias formation during pre-training.

Summary updated

BibTeX
@inproceedings{chen2024mitigating,
  author = {Peng Chen and Xiao-Yu Guo and Yuan-Fang Li and Xiaowang Zhang and Zhiyong Feng},
  title = {{Mitigating Language Bias of LMMs in Social Intelligence Understanding with Virtual Counterfactual Calibration}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.77},
}
Figure