Abstract
Developing cultural adaptation methods is important, which can improve the model performance on the low-resource ones and provide more equitable opportunities for everyone to benefit from advanced technology. Past methods primarily focused on multilingual and multimodal capabilities, and the improvement of multicultural competence is still an unexplored problem. This is largely due to the difficulty of data scarcity and expensive annotation. In this paper, we navigate this uncharted territory by leveraging high-resource cultures to facilitate comprehension of low-resource ones. We first introduce an annotation-free method for cultural-concept adaptation and construct a concept mapping set. To facilitate the model's comprehension of cultural-concept mappings, we propose a new multimodal data augmentation called CultureMixup. This approach employs a three-tier code-switching strategy on textual sentences. Additionally, it uses a cultural concept-based mixup method for the images. This combination effectively generates new data instances across culture, phrase, word, and image levels. For visually grounded reasoning across languages and cultures, experimental results on five languages show that our method consistently improves performance for four existing multilingual and multimodal models on both zero-shot and few-shot settings.
Introduction
Large-scale multimodal models have demonstrated impressive capabilities in vision-language tasks but often exhibit a significant "Western-centric" bias. While these models can easily identify common Western objects like a "violin," they frequently struggle with culturally specific concepts from low-resource regions, such as the Chinese "Erhu." This performance gap stems from two primary issues: the scarcity of labeled data for diverse cultural concepts and the high cost of manual cross-cultural annotation. This research addresses these challenges by introducing an annotation-free framework that leverages high-resource cultural knowledge to bridge the gap to low-resource cultures.
Methodology
The proposed solution consists of two main components: a Cultural Adaptation Graph and the CultureMixup augmentation strategy.
Cultural Adaptation Graph
To avoid expensive manual labeling, the authors utilize semantic dictionaries like ConceptNet and WordNet to build a graph of relationships. By traversing hypernyms, hyponyms, and synonyms, the system identifies "bridge" concepts. For instance, the "Erhu" is linked to its hypernym "bowed string instrument," which is also a hypernym for the "violin." This allows the model to understand the Erhu through its relationship to the more familiar violin.
CultureMixup
To train models on these mappings, the authors developed CultureMixup, a multimodal data augmentation technique. As shown in Figure 2, this method involves:
- Three-tier Code-switching (Text): English sentences are modified at the cultural level (e.g., swapping "violin" for "Erhu"), phrase level (translating spatial phrases like "on the left"), and word level (randomly translating individual words).
- Concept-Based Mixup (Images): High-resource objects in images are programmatically replaced with low-resource equivalents using object detection and image blending techniques.
Experimental Setup
The framework was evaluated using the following datasets:
- NLVR2: The primary English training set for visual reasoning.
- MaRVL (Multicultural Reasoning and Visual Language): A benchmark covering five languages and cultures: Indonesian (ID), Swahili (SW), Tamil (TA), Turkish (TR), and Mandarin Chinese (ZH).
- New Cultural Concept Set: A validated set of cultural concepts across 10 categories (e.g., food, festivals, tools) for the target cultures.
Results
The method was tested on four major multimodal models: mUNITER, xUNITER, UC2, and M3P. As detailed in Figure 4, the approach yielded consistent performance gains across all five languages in zero-shot settings.
Key findings include:
- Quantitative Improvements: For the mUNITER model, the average score on the MaRVL benchmark increased from 53.51% to 57.31%.
- Reduced Disparity: The gap between "Translate-Test" and "Cross-lingual Transfer" was significantly narrowed, indicating that the models are learning genuine cultural concepts rather than relying solely on translation.
Discussion and Conclusion
This work demonstrates that "cultural" code-switching—swapping concepts rather than just languages—forces models to align diverse cultural representations. By using shared hypernyms as bridges, knowledge can be effectively transferred from high-resource to low-resource domains. Furthermore, the authors show that adjusting sampling probabilities during augmentation allows for controllable bias mitigation, surgically improving performance for specific cultures. However, models still struggle with complex spatial reasoning, and future work is needed to address more abstract "cultural common sense" and social norms. Despite these limitations, the proposed framework offers a scalable, annotation-free path toward more culturally inclusive multimodal AI.
Summary updated