Culturally-aware Image Captioning

Youngsik YunIJCAI 2024

Abstract

Current Vision-Language Pre-trained models (VLPs) often suffer from geographical and demographic biases due to their reliance on predominantly Western-centric training data. This paper introduces the Culturally-aware Image Caption (CIC) framework, a novel pipeline designed to mitigate the "erasure" of cultural specificity in AI-generated descriptions. By utilizing targeted Visual Question Answering (VQA) to probe for specific cultural elements and Large Language Models (LLMs) to synthesize these details, the CIC framework moves beyond generic labeling toward more accurate, culturally-rich representations.

1. Introduction

Most state-of-the-art Vision-Language models are trained on massive, web-crawled datasets such as LAION-400M. While these datasets provide the scale necessary for high performance, they are heavily skewed toward Western contexts. Consequently, when these models encounter images from minority groups or non-Western regions, they often fail to recognize specific cultural nuances. A Korean hanbok might be labeled simply as "traditional attire," or a specific religious ritual might be described in overly generic terms. This research seeks to address this "generic trap" by enabling models to "see" and articulate cultural details without requiring the massive, often unavailable, training sets for every individual ethnic group.

2. The CIC Framework

The proposed Culturally-aware Image Caption (CIC) framework replaces the standard one-pass captioning approach with a structured, multi-step pipeline:

  1. Initial Screening: The process begins with a base caption generated by a model like BLIP-2. If the system detects keywords related to cultural categories, the specialized cultural awareness module is triggered.
  2. Targeted Visual Question Answering (VQA): To extract granular details, the framework "probes" the image using questions based on five sociological categories defined by Halpern (1955): Architecture, Clothing, Food & Drinks, Dance & Music, and Religion. For example, if an image contains a building, the system asks, "What is the architecture style of the buildings in this image?"
  3. LLM Synthesis: The initial caption and the specific cultural elements extracted via VQA are fed into an LLM. The LLM then synthesizes this information into a final caption that correctly identifies specific items—transforming a generic description into one that recognizes a hanbok or a specific temple style.

3. Data and Evaluation Challenges

The research highlights a critical flaw in current AI evaluation: the datasets and metrics themselves are biased.

  • Dataset Bias: LAION-400M is identified as a primary source of the Western-centric bias that leads models to overlook non-Western cultural markers.
  • Metric Limitations: Standard automated metrics like BLEU and ROUGE are poorly suited for this task. These metrics rely on word-matching and do not adequately reward the inclusion of specific cultural terminology over generic descriptions. Furthermore, CLIPScore—often used for image-text alignment—is trained on the same biased data, making it "blind" to the improvements the CIC framework aims to achieve.
  • Human-Centric Evaluation: To overcome these limitations, the study recruited participants from specific cultural regions to perform qualitative evaluations. These human evaluators consistently preferred the CIC framework over standard VLP models for its accuracy in depicting cultural information.

4. Key Insights

VQA as a Cultural Probe

A significant finding of this work is that AI models often "know" cultural details but fail to prioritize them in general captions. By using VQA to force the model to attend to specific categories (like clothing or food), the framework effectively extracts latent knowledge that would otherwise be ignored.

Sociological Grounding

By adopting Halpern’s five categories of culture, the framework moves away from ad-hoc prompting. This structured, sociologically-grounded method ensures a more comprehensive understanding of the visual scene, moving the needle from mere identification to cultural appreciation.

5. Limitations and Future Work

While the CIC framework represents a significant step forward, several hurdles remain. First, there is a risk of stereotyping; the framework currently lacks a built-in mechanism to filter out biases or harmful stereotypes inherent in the underlying LLMs. Second, the reliance on human evaluation highlights the urgent need for new, culturally-sensitive automated benchmarks.

Future research will focus on developing ethical standards and involving diverse cultural perspectives to ensure that AI-generated captions are respectful, accurate, and truly representative of global diversity.

Summary updated

BibTeX
@inproceedings{yun2024culturally,
  author = {Youngsik Yun},
  title = {{Culturally-aware Image Captioning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure