Cultural Concept Adaptation on Multimodal Reasoning

Zhi Li, Yin ZhangEMNLP 2023 · OpenReview

Abstract

Developing cultural adaptation methods is important, which can improve the model performance on the low-resource ones and provide more equitable opportunities for everyone to benefit from advanced technology. Past methods primarily focused on multilingual and multimodal capabilities, and the improvement of multicultural competence is still an unexplored problem. This is largely due to the difficulty of data scarcity and expensive annotation. In this paper, we navigate this uncharted territory by leveraging high-resource cultures to facilitate comprehension of low-resource ones. We first introduce an annotation-free method for cultural-concept adaptation and construct a concept mapping set. To facilitate the model's comprehension of cultural-concept mappings, we propose a new multimodal data augmentation called CultureMixup. This approach employs a three-tier code-switching strategy on textual sentences. Additionally, it uses a cultural concept-based mixup method for the images. This combination effectively generates new data instances across culture, phrase, word, and image levels. For visually grounded reasoning across languages and cultures, experimental results on five languages show that our method consistently improves performance for four existing multilingual and multimodal models on both zero-shot and few-shot settings.

Introduction

Large-scale multimodal models have demonstrated impressive capabilities in vision-language tasks but often exhibit a significant "Western-centric" bias. While these models can easily identify common Western objects like a "violin," they frequently struggle with culturally specific concepts from low-resource regions, such as the Chinese "Erhu." This performance gap stems from two primary issues: the scarcity of labeled data for diverse cultural concepts and the high cost of manual cross-cultural annotation. This research addresses these challenges by introducing an annotation-free framework that leverages high-resource cultural knowledge to bridge the gap to low-resource cultures.

Methodology

The proposed solution consists of two main components: a Cultural Adaptation Graph and the CultureMixup augmentation strategy.

Cultural Adaptation Graph

To avoid expensive manual labeling, the authors utilize semantic dictionaries like ConceptNet and WordNet to build a graph of relationships. By traversing hypernyms, hyponyms, and synonyms, the system identifies "bridge" concepts. For instance, the "Erhu" is linked to its hypernym "bowed string instrument," which is also a hypernym for the "violin." This allows the model to understand the Erhu through its relationship to the more familiar violin.

CultureMixup

To train models on these mappings, the authors developed CultureMixup, a multimodal data augmentation technique. As shown in Figure 2, this method involves:

  • Three-tier Code-switching (Text): English sentences are modified at the cultural level (e.g., swapping "violin" for "Erhu"), phrase level (translating spatial phrases like "on the left"), and word level (randomly translating individual words).
  • Concept-Based Mixup (Images): High-resource objects in images are programmatically replaced with low-resource equivalents using object detection and image blending techniques.
Overview of the CultureMixup framework, illustrating the three-tier code-switching strategy and cultural concept mapping.

Experimental Setup

The framework was evaluated using the following datasets:

  • NLVR2: The primary English training set for visual reasoning.
  • MaRVL (Multicultural Reasoning and Visual Language): A benchmark covering five languages and cultures: Indonesian (ID), Swahili (SW), Tamil (TA), Turkish (TR), and Mandarin Chinese (ZH).
  • New Cultural Concept Set: A validated set of cultural concepts across 10 categories (e.g., food, festivals, tools) for the target cultures.

Results

The method was tested on four major multimodal models: mUNITER, xUNITER, UC2, and M3P. As detailed in Figure 4, the approach yielded consistent performance gains across all five languages in zero-shot settings.

Zero-shot performance results on NLVR2 and MaRVL benchmarks, comparing the proposed method against baseline models.

Key findings include:

  • Quantitative Improvements: For the mUNITER model, the average score on the MaRVL benchmark increased from 53.51% to 57.31%.
  • Reduced Disparity: The gap between "Translate-Test" and "Cross-lingual Transfer" was significantly narrowed, indicating that the models are learning genuine cultural concepts rather than relying solely on translation.

Discussion and Conclusion

This work demonstrates that "cultural" code-switching—swapping concepts rather than just languages—forces models to align diverse cultural representations. By using shared hypernyms as bridges, knowledge can be effectively transferred from high-resource to low-resource domains. Furthermore, the authors show that adjusting sampling probabilities during augmentation allows for controllable bias mitigation, surgically improving performance for specific cultures. However, models still struggle with complex spatial reasoning, and future work is needed to address more abstract "cultural common sense" and social norms. Despite these limitations, the proposed framework offers a scalable, annotation-free path toward more culturally inclusive multimodal AI.

Summary updated

Abstract

Developing cultural adaptation methods is important, which can improve the model performance on the low-resource ones and provide more equitable opportunities for everyone to benefit from advanced technology. Past methods primarily focused on multilingual and multimodal capabilities, and the improvement of multicultural competence is still an unexplored problem. This is largely due to the difficulty of data scarcity and expensive annotation. In this paper, we navigate this uncharted territory by leveraging high-resource cultures to facilitate comprehension of low-resource ones. We first introduce an annotation-free method for cultural-concept adaptation and construct a concept mapping set. To facilitate the model's comprehension of cultural-concept mappings, we propose a new multimodal data augmentation called CultureMixup. This approach employs a three-tier code-switching strategy on textual sentences. Additionally, it uses a cultural concept-based mixup method for the images. This combination effectively generates new data instances across culture, phrase, word, and image levels. For visually grounded reasoning across languages and cultures, experimental results on five languages show that our method consistently improves performance for four existing multilingual and multimodal models on both zero-shot and few-shot settings.
BibTeX
@inproceedings{li2023cultural,
  author = {Zhi Li and Yin Zhang},
  title = {{Cultural Concept Adaptation on Multimodal Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=xyvTFX7hDs},
}
Figure