Abstract
Large Multimodal Models (LMMs) often struggle with social intelligence tasks—understanding human intentions, emotions, and interactions—because they rely heavily on text-based "common sense" priors rather than visual evidence. This paper introduces DCVC (Distribution Calibration network with Virtual Counterfactual data augmentation), a framework that uses causal modeling and virtual "what-if" scenarios to calibrate model outputs. By addressing the "back-door" path of language bias, DCVC significantly boosts performance on social reasoning benchmarks like Social-IQ-2.0 without requiring expensive manual data labeling.
Introduction: The Language Bias Problem
Social intelligence is a critical frontier for AI. While LMMs like Video-LLaVA are powerful, they often exhibit a significant language bias. Because these models are pre-trained on vast text corpora, they develop strong internal priors. When presented with a video and a question (e.g., "How does not knowing the route make the woman feel?"), the model might ignore the visual context and select the most statistically likely text answer based on the question alone, leading to incorrect reasoning.
This "spurious correlation" between internal text knowledge and provided options often results in a "stochastic parrot" effect in social settings, where the model's prediction with video is nearly identical to its prediction without it.
Methodology: The DCVC Framework
The authors propose the DCVC framework to mitigate these biases through three core components:
- Structural Causal Model (SCM): Causal theory is used to identify how bias enters the system, specifically identifying internal knowledge as a "back-door" path that bypasses visual/audio context.
- Output Distribution Calibration (DC): A lightweight calibration network is added to re-weight the LMM's raw predictions against language priors, ensuring the final answer is grounded in multimodal context.
- Virtual Counterfactual Data Augmentation (VCDA): To train the calibration network without expensive video editing, the authors create "virtual" samples. By applying Gumbel noise to output distributions, they simulate how the model would react if visual context were different, allowing it to learn to ignore its biases.
Experimental Evaluation
The framework was evaluated on three major benchmarks: Social-IQ-2.0 (social intelligence), DeSIQ-2.0 (debiased social intelligence), and NExT-QA (general video QA).
Key Results
- Performance Gains: On Social-IQ-2.0, adding DCVC to Video-LLaVA increased its score from 61.06% to 78.32%—a 17-point improvement.
- Superiority over Fine-tuning: On DeSIQ-2.0, Video-LLaVA + DCVC (97.04%) outperformed models specifically fine-tuned on that dataset.
- Generalizability: The performance boost on NExT-QA suggests that language bias is a universal hurdle for multimodal models beyond just social intelligence.
Conclusion
DCVC demonstrates that calibrating model outputs is often more effective and efficient than massive retraining. By performing counterfactual reasoning in the "probability space" rather than the "pixel space," the framework preserves beneficial linguistic priors while filtering out harmful social stereotypes. Future work will explore scaling the method to larger models and investigating the root causes of bias formation during pre-training.
Summary updated