The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning

Longju Bai, Angana Borah, Oana Ignat, Rada Mihalcea

TL;DR

Current Large Multimodal Models (LMMs) suffer from a distinct Western-centric bias, frequently missing or misinterpreting cultural nuances in images. To address this, the authors introduce MosAIC, a multi-agent framework where LMMs embodying distinct cultural personas (China, India, and Romania) collaborate via a question-and-answer dialogue to generate culturally enriched image captions. This collaborative approach significantly outperforms single-agent baselines in capturing cultural information without requiring expensive fine-tuning.

What Problem Does This Paper Address?

While LMMs excel at general vision-language tasks, their training data is predominantly Western-centric. This creates an "algorithmic monoculture" that struggles with cross-cultural contexts. A single model often fails to identify country-specific foods, clothing, or rituals, or describes them using vague, Westernized terms. Furthermore, standard evaluation metrics like CLIPScore focus heavily on direct text-to-image alignment, penalizing the broader cultural context (like the history or significance of a ritual) that isn't explicitly visible in the pixels.

Methods

Instead of treating image captioning as a single-model generation task, the authors frame it as a "social task" rooted in collective intelligence. They propose the MosAIC framework, which deploys five distinct LMM agents (simulated using LLaVA-1.5 13B):

  • Moderator Agent: Generates initial questions about the image and guides the conversation toward cultural elements.
  • Social Agents (3): Assigned specific cultural personas—China (C), India (I), and Romania (R)—and prompted to be curious and collaborative.
  • Summarizer Agent: Compiles the final, culturally enriched caption from the social agents' discussions.
Figure 3: Multi-Agent Interaction Model. The Modera- tor presents questions to the Social agents, who engage in three conversation rounds. The Summarizer creates the final image caption by compiling the conversation summaries from the Social agents.

The agents interact over three distinct conversation rounds. Initially, the social agents independently describe the image to avoid bias. In subsequent rounds, they review their peers' responses, answer questions from their own cultural perspectives, and ask new questions.

To evaluate the results, the authors also introduce a length-invariant Cultural Information metric. Unlike previous ratio-based metrics that penalize longer captions, this metric counts unique cultural words by matching generation outputs against an expanded cultural commonsense knowledge base.

Datasets

The framework was evaluated using images from three geographically diverse datasets:

  1. GeoDE: A dataset designed for geo-diverse object recognition.
  2. GD-VCR: A geo-diverse visual commonsense reasoning dataset.
  3. CVQA: A culturally diverse multilingual visual question-answering benchmark.

Using these, the authors generated a new dataset of 2,832 culturally enriched English captions. They also gathered a human baseline benchmark consisting of captions from nine native annotators (three per country) across a subset of 75 images.

Key Insights and Ideas

  • Interaction Trumps Fine-Tuning: Multi-agent emergent collaboration proved more effective at introducing cultural context than compute-intensive fine-tuning on culture-specific datasets (like Wikipedia's WIT).
  • LMMs Over-Index on Culture, Humans on Correctness: Interestingly, LMMs naturally generate more cultural words than human annotators. This is likely due to statistical learning from diverse training data and a lack of personal context, whereas humans favor concise, direct descriptions.
  • Simple Prompts Win: When testing prompting strategies (Simple, Multilingual, Anthropological, and Chain-of-Thought), Chain-of-Thought (CoT) performed best. Complex anthropological framing actually introduced more hallucinations, showing that LMMs still need basic, structured guidance rather than abstract academic framing.

Key Results

MosAIC significantly outperformed both single-agent baselines (BLIP-2, LLaVA-13b) and humans on Completeness and Cultural Information, while maintaining competitive text-to-image Alignment.

Figure 5: Our interaction-based model, MosAIC, sur- passes non-interaction models and Humans on Com- pleteness and Cultural Info while performing on par with the other models in Alignment. For clarity, the Alignment and Completeness scores are normalized to a [0,1] scale, whereas the Cultural Info score ranges from 0 to the total number of words in a caption.
  • Cultural Info Score: MosAIC achieved a score of 26.01, nearly doubling the single-agent LLaVA-13b baseline (14.55) and outperforming the expert human baseline (15.57).
  • Completeness: MosAIC scored 0.41 compared to LLaVA-13b's 0.28.
  • Efficiency: While a fine-tuned single LLaVA model required 54 hours of total compute (training + inference) on an NVIDIA A100 GPU, the zero-shot MosAIC framework achieved superior cultural depth in just 47 hours of pure inference.

Limitations and Future Work

The primary drawback of the multi-agent approach is compound hallucination. Because the agents feed off each other's text histories, an error or misidentification by one agent can amplify through the conversation rounds. Consequently, MosAIC scored lower on caption correctness (60.2%) than the single-agent LLaVA baseline (64.56%).

Additionally, the authors note that using "countries" as a proxy for culture is an oversimplification that ignores regional variations and individual subcultures. Future work will focus on integrating state-of-the-art object recognition systems to anchor the agents in physical reality (reducing hallucinations) and expanding the framework to encompass a broader, more diverse set of global cultures.

Summary updated

Abstract

Large Multimodal Models (LMMs) exhibit impressive performance across various multimodal tasks. However, their effectiveness in cross-cultural contexts remains limited due to the predominantly Western-centric nature of most data and models. Conversely, multi-agent models have shown significant capability in solving complex tasks. Our study evaluates the collective performance of LMMs in a multi-agent interaction setting for the novel task of cultural image captioning. Our contributions are as follows: (1) We introduce MosAIC, a Multi-Agent framework to enhance cross-cultural Image Captioning using LMMs with distinct cultural personas; (2) We provide a dataset of culturally enriched image captions in English for images from China, India, and Romania across three datasets: GeoDE, GD-VCR, CVQA; (3) We propose a culture-adaptable metric for evaluating cultural information within image captions; and (4) We show that the multi-agent interaction outperforms single-agent models across different metrics, and offer valuable insights for future research. Our dataset and models can be accessed at https://github.com/MichiganNLP/MosAIC.
BibTeX
@inproceedings{bai2025power,
  author = {Longju Bai and Angana Borah and Oana Ignat and Rada Mihalcea},
  title = {{The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.152},
}
Figure