CIC: A Framework for Culturally-Aware Image Captioning

Youngsik Yun, Jihie KimIJCAI 2024

Abstract

While modern Vision-Language Pre-trained models (VLPs) excel at general scene description, they often exhibit "cultural blindness," failing to recognize and describe specific cultural nuances such as traditional attire, architectural styles, or religious customs. This paper introduces the Culturally-aware Image Captioning (CIC) framework, a multi-stage pipeline designed to bridge this gap. By leveraging targeted Visual Question Answering (VQA) and the reasoning capabilities of Large Language Models (LLMs), CIC generates captions that are both visually grounded and culturally rich.

Introduction

Standard image captioning models like BLIP or GIT typically produce generic descriptions (e.g., "people sitting at a table") that overlook the diverse "ways of life" represented globally. This cultural blindness stems from Western-centric training datasets and evaluation metrics like BLEU or CIDEr, which reward generic accuracy over cultural specificity. For AI to be truly inclusive, it must recognize elements like a Japanese kimono or South Asian kurta pyjama as distinct entities rather than generic clothing.

Comparison of generic captions versus culturally-aware descriptions across different cultural groups.

Methodology

The CIC framework operates through a three-phase pipeline to ensure cultural elements are accurately captured without introducing hallucinations:

  1. Cultural Question Generation: The authors identified five key categories: Architecture, Clothing, Food & Drink, Dance & Music, and Religion. A bank of 1,054 questions was developed and refined through clustering to ensure precision.
  2. Visual Extraction via VQA: To ground the description in reality, the system extracts keywords from a baseline caption. If "people" are mentioned, it triggers VQA questions regarding "Clothing." The BLIP2 model is then used to answer these specific cultural queries.
  3. LLM Synthesis: Finally, the baseline caption and VQA results (e.g., "Clothing: kimono") are fed into ChatGPT. Using a specialized prompt, the LLM synthesizes these inputs into a cohesive, culturally-aware sentence.
The CIC framework pipeline: from image to culturally-aware caption via VQA and LLM synthesis.

Datasets and Evaluation

The framework was evaluated using the GD-VCR (Geo-diverse Visual Commonsense Reasoning) dataset, which contains images from four major cultural groups: West, East Asia, South Asia, and Africa. The authors also introduced the Culture Noise Rate (CNR), a metric measuring the density of cultural terminology in a caption.

Key Results

  • Human Preference: In a survey of 45 participants from the represented regions, CIC significantly outperformed baselines. In the "West" group, CIC achieved a cultural quality score of 0.62, compared to 0.20 for BLIP2.
  • Quantitative Improvement: CIC generated approximately 17% more cultural words than baseline models as measured by CNR.
  • Enhanced Alignment: CIC achieved the highest CLIPScore in three out of four cultural groups, indicating that adding cultural detail improves overall image-text alignment rather than detracting from it.

Discussion and Limitations

A core insight of this work is the use of LLMs as "cultural synthesizers." While VLPs are proficient at visual detection, they lack the contextual knowledge to label cultural specifics accurately. By using VQA as a grounding bridge, CIC allows LLMs to apply cultural knowledge to raw visual data safely.

However, limitations remain. The current framework is restricted to five categories and may miss nuances related to modern urban styles or subtle social behaviors. Furthermore, the evaluation was conducted on a relatively small set of 329 images. Future work aims to expand the categorization and scale the framework to larger, more diverse benchmarks to capture the full complexity of human culture.

Summary updated

BibTeX
@inproceedings{yun2024cic,
  author = {Youngsik Yun and Jihie Kim},
  title = {{CIC: A Framework for Culturally-Aware Image Captioning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure