Abstract
Image captioning models often rely on spurious correlations found in training data, leading to "hallucinations" where objects are predicted based on context rather than visual evidence. We present CIIC (Causal Inference based Image Captioning), a framework that leverages Structural Causal Models and causal intervention to deconfound visual and linguistic features. By applying the do-calculus at both encoding and decoding stages, CIIC learns to describe images based on true causal relationships, achieving state-of-the-art performance on MSCOCO while significantly reducing object bias.
Introduction: The Challenge of Spurious Correlations
Standard transformer-based image captioning models excel at pattern recognition but frequently fall prey to statistical coincidences. These "spurious correlations" manifest in two primary ways:
- Visual Confounders: Co-occurring objects (e.g., "cake" and "fork") lead object detectors to produce entangled features, where a "fork" feature is heavily biased by the presence of "cake."
- Linguistic Confounders: Language models learn that certain words frequently follow others, causing them to predict a "fork" simply because "cake" was mentioned, even if the image actually contains a "spoon."
These biases hinder generalization and lead to harmful hallucinations, where the model prioritizes dataset shortcuts over actual visual input.
Methodology: The CIIC Framework
The CIIC framework redefines image captioning as a causal inference task. Using Structural Causal Models (SCMs) and backdoor adjustment, the model "blocks" the influence of confounders.
Interventional Object Detector (IOD)
The IOD replaces standard detectors like Faster R-CNN. It uses causal intervention to disentangle object features from their surrounding context. By simulating the appearance of an object across a diverse range of contexts, the IOD ensures that the extracted features represent the object itself rather than its environment.
Interventional Transformer Decoder (ITD)
During caption generation, the ITD deconfounds both visual and linguistic streams. It utilizes a "confounder dictionary"—an approximation of global context distributions—to ensure that each predicted word is causally linked to specific image regions. This prevents the model from relying on common word pairings (e.g., cake $\rightarrow$ fork) when the visual evidence suggests otherwise.
Unaligned Feature Transformer (UFT)
To maintain high discriminative power while ensuring causal accuracy, the UFT encoder fuses deconfounded IOD features with traditional bottom-up features. The entire system is optimized using the Normalized Weighted Geometric Mean (NWGM) approximation, making the complex causal math computationally efficient for large-scale training.
Experimental Results
The CIIC model was evaluated on the MSCOCO dataset, showing significant improvements in both accuracy and reliability.
Performance and Bias Reduction
- Accuracy: CIIC achieved a state-of-the-art CIDEr score of 133.1 on the MSCOCO Karpathy test split.
- Hallucination Mitigation: Using the CHAIR (Caption Hallucination Assessment in Image Recall) metric, CIIC demonstrated a substantial reduction in object bias. The CHAIRs score dropped from 12.1 (baseline Transformer) to 7.7, indicating that the model is far less likely to hallucinate contextually likely but visually absent objects.
Conclusion
By integrating causal inference into the image captioning pipeline, CIIC moves beyond simple likelihood estimation toward a more robust understanding of visual scenes. While the framework currently relies on a pre-defined confounder dictionary, it sets a new precedent for addressing dataset bias in vision-language tasks. Future work may explore more dynamic ways to model global context and extend these causal principles to tasks like Visual Question Answering (VQA).
Summary updated