Visual Question Answering for Peruvian Cuisine in Regional Spanish

Mariana Risco CosavalenteAAAI 2025 · DOI

Abstract

Visual Question Answering (VQA) systems often struggle with cultural and linguistic nuances outside of dominant English-language datasets. This paper presents a methodology for a specialized VQA system focused on Peruvian cuisine, utilizing regional Spanish. By fine-tuning multimodal models like LLaVA-1.5 on a culturally specific dataset, the project aims to bridge the "cultural gap" in AI, providing a more inclusive tool for one of the world's premier culinary destinations.

Introduction: The Cultural Gap in AI

While Visual Question Answering (VQA)—the ability of an AI to answer questions about an image—has advanced rapidly, it suffers from a significant "cultural gap." Most state-of-the-art models are trained primarily on English-language datasets, which often fail to capture the linguistic diversity and cultural specifics of the Spanish-speaking world.

In Peru, gastronomy is a cornerstone of national identity. However, the complexity of Peruvian food is reflected in its language; a dish might have different names, ingredients, or historical contexts depending on whether it originates from the Coast, the Sierra (highlands), or the Selva (jungle). Standard AI models often lack this regional granularity, resulting in poor performance—historically around 32.5% accuracy in Spanish food-related tasks. This research addresses the need for AI that is both linguistically inclusive and culturally aware.

Methodology and System Architecture

The proposed pipeline combines advanced computer vision with regional natural language processing to ensure high accuracy in identifying and describing complex dishes.

The proposed VQA pipeline, illustrating the flow from input (image and question) through the model to the final output.

Model Selection

The project utilizes LLaVA-1.5, a state-of-the-art large multimodal model capable of processing both visual and textual inputs. To enhance the linguistic component, the system integrates GPT-2 Large models from the "MarIA" family, which were specifically trained on massive Spanish corpora to better handle the nuances of the language.

Processing Techniques

  • Named Entity Recognition (NER): The system uses NER to identify and extract critical entities such as specific dish names, ingredients, and regional terms from textual descriptions.
  • Zero-Shot Prompting: To handle dishes not present in the training data, the author employs zero-shot prompting, leveraging the models' pre-existing general knowledge to make educated inferences about unknown culinary items.
  • Data Augmentation: Standard techniques like rotation, scaling, and noise injection are used to ensure the model can recognize a dish regardless of photo quality or lighting—essential for real-world tourist photos taken in diverse environments.

Dataset Construction: PeruFoodNet

A central pillar of this research is the creation of a multi-source, culturally grounded dataset. The project introduces PeruFoodNet, which consists of 4,000 images across 40 traditional dish categories. To ensure the dataset reflects real-world usage, the author employed several collection strategies:

  • Social Media Scraping: Using Selenium to harvest images and text from hashtags like #ceviche and #anticucho.
  • Google Maps API: Extracting reviews and photos from regional restaurants to capture how locals actually describe their food.
  • Culinary Literature: Incorporating authoritative books to provide historical and cultural context.
  • Regional Annotation: Crucially, the dataset is annotated in collaboration with linguistics students from the Coast, Sierra, and Selva to ensure regional naming variations are accurately captured.

Key Insights and Evaluation

The paper argues that VQA is not merely about identifying objects; it is about understanding the "regional linguistic peculiarities" that define a culture. By involving Peruvians from diverse and underrepresented regions in the dataset construction, the project moves away from a "one-size-fits-all" Spanish toward a more democratic AI.

Furthermore, the author emphasizes that automated metrics (like F1 and Perplexity) are insufficient for cultural tasks. The project incorporates Human Evaluation from culinary experts and senior citizens to validate the "correctness" of the AI’s cultural nuances and historical grounding.

Conclusion and Future Directions

As a project proposal presented at the AAAI-25 Undergraduate Consortium, this work highlights the significant room for improvement over baseline Spanish VQA models. Future work will focus on scaling the dataset and addressing the risk of "hallucinations" in Large Language Models. The ultimate goal is the deployment of a mobile application that can serve as a cultural bridge for tourists, requiring further research into the latency and computational requirements of running multimodal models on mobile devices.

Summary updated

BibTeX
@inproceedings{cosavalente2025visual,
  author = {Mariana Risco Cosavalente},
  title = {{Visual Question Answering for Peruvian Cuisine in Regional Spanish}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35339},
}
Figure