TL;DR
Current Large Multimodal Models (LMMs) suffer from a distinct Western-centric bias, frequently missing or misinterpreting cultural nuances in images. To address this, the authors introduce MosAIC, a multi-agent framework where LMMs embodying distinct cultural personas (China, India, and Romania) collaborate via a question-and-answer dialogue to generate culturally enriched image captions. This collaborative approach significantly outperforms single-agent baselines in capturing cultural information without requiring expensive fine-tuning.
What Problem Does This Paper Address?
While LMMs excel at general vision-language tasks, their training data is predominantly Western-centric. This creates an "algorithmic monoculture" that struggles with cross-cultural contexts. A single model often fails to identify country-specific foods, clothing, or rituals, or describes them using vague, Westernized terms. Furthermore, standard evaluation metrics like CLIPScore focus heavily on direct text-to-image alignment, penalizing the broader cultural context (like the history or significance of a ritual) that isn't explicitly visible in the pixels.
Methods
Instead of treating image captioning as a single-model generation task, the authors frame it as a "social task" rooted in collective intelligence. They propose the MosAIC framework, which deploys five distinct LMM agents (simulated using LLaVA-1.5 13B):
- Moderator Agent: Generates initial questions about the image and guides the conversation toward cultural elements.
- Social Agents (3): Assigned specific cultural personas—China (C), India (I), and Romania (R)—and prompted to be curious and collaborative.
- Summarizer Agent: Compiles the final, culturally enriched caption from the social agents' discussions.
The agents interact over three distinct conversation rounds. Initially, the social agents independently describe the image to avoid bias. In subsequent rounds, they review their peers' responses, answer questions from their own cultural perspectives, and ask new questions.
To evaluate the results, the authors also introduce a length-invariant Cultural Information metric. Unlike previous ratio-based metrics that penalize longer captions, this metric counts unique cultural words by matching generation outputs against an expanded cultural commonsense knowledge base.
Datasets
The framework was evaluated using images from three geographically diverse datasets:
- GeoDE: A dataset designed for geo-diverse object recognition.
- GD-VCR: A geo-diverse visual commonsense reasoning dataset.
- CVQA: A culturally diverse multilingual visual question-answering benchmark.
Using these, the authors generated a new dataset of 2,832 culturally enriched English captions. They also gathered a human baseline benchmark consisting of captions from nine native annotators (three per country) across a subset of 75 images.
Key Insights and Ideas
- Interaction Trumps Fine-Tuning: Multi-agent emergent collaboration proved more effective at introducing cultural context than compute-intensive fine-tuning on culture-specific datasets (like Wikipedia's WIT).
- LMMs Over-Index on Culture, Humans on Correctness: Interestingly, LMMs naturally generate more cultural words than human annotators. This is likely due to statistical learning from diverse training data and a lack of personal context, whereas humans favor concise, direct descriptions.
- Simple Prompts Win: When testing prompting strategies (Simple, Multilingual, Anthropological, and Chain-of-Thought), Chain-of-Thought (CoT) performed best. Complex anthropological framing actually introduced more hallucinations, showing that LMMs still need basic, structured guidance rather than abstract academic framing.
Key Results
MosAIC significantly outperformed both single-agent baselines (BLIP-2, LLaVA-13b) and humans on Completeness and Cultural Information, while maintaining competitive text-to-image Alignment.
- Cultural Info Score: MosAIC achieved a score of 26.01, nearly doubling the single-agent LLaVA-13b baseline (14.55) and outperforming the expert human baseline (15.57).
- Completeness: MosAIC scored 0.41 compared to LLaVA-13b's 0.28.
- Efficiency: While a fine-tuned single LLaVA model required 54 hours of total compute (training + inference) on an NVIDIA A100 GPU, the zero-shot MosAIC framework achieved superior cultural depth in just 47 hours of pure inference.
Limitations and Future Work
The primary drawback of the multi-agent approach is compound hallucination. Because the agents feed off each other's text histories, an error or misidentification by one agent can amplify through the conversation rounds. Consequently, MosAIC scored lower on caption correctness (60.2%) than the single-agent LLaVA baseline (64.56%).
Additionally, the authors note that using "countries" as a proxy for culture is an oversimplification that ignores regional variations and individual subcultures. Future work will focus on integrating state-of-the-art object recognition systems to anchor the agents in physical reality (reducing hallucinations) and expanding the framework to encompass a broader, more diverse set of global cultures.
Summary updated