Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic

Fakhraddin Alwajih, Gagan Bhatia, Muhammad Abdul-MageedACL 2024

TL;DR

This paper introduces Dallah, a novel multimodal large language model (MLLM) specifically designed to understand and generate content in various Arabic dialects, bridging a significant gap in current MLLM capabilities. By leveraging a unique translation and filtering pipeline and fine-tuning on dialect-specific data, Dallah achieves state-of-the-art performance on Arabic MLLM benchmarks, demonstrating robust dialect authenticity and content accuracy.

What Problem Does This Paper Address?

The field of multimodal large language models (MLLMs) has seen rapid advancements, primarily driven by the availability of high-quality English datasets. However, this progress has largely excluded other languages, particularly those with rich dialectal variations like Arabic. Standard NLP models often struggle with these dialects, and existing MLLMs are not equipped to handle the nuances of Arabic's linguistic diversity, leading to a gap in equitable AI development. This paper aims to address this by creating an MLLM that can effectively process and generate content across different Arabic dialects, preserving linguistic heritage and enabling broader accessibility.

Methods

Dallah is built upon the LLaVA1.5 architecture, integrating a CLIP-Large vision encoder and the AraLLaMA language model. The core innovation lies in a novel "translate-and-filter" pipeline for creating Arabic multimodal datasets. This pipeline uses Google Translate for initial translation from English to Arabic, followed by a back-translation and similarity scoring stage using a sentence embedding model to filter out low-quality translations. This ensures the fine-tuning data is both accurate and representative of Arabic's linguistic diversity. Dallah's training involves three stages: pre-training on a combined Arabic and English dataset, visual instruction fine-tuning on MSA and Arabic LLaVA-Instruct data, and finally, dialectal instruction fine-tuning on six major Arabic dialects.

Datasets

  • Training Data:
    • LLaVA-Pretrain (MAS Arabic and English)
    • LLaVA-Instruct (English and translated Arabic MSA)
    • Dialectal data derived from LLaVA-Instruct, translated by native speakers for six Arabic dialects (Egypt, Mauritania, Morocco, Palestine, Saudi Arabia, Yemen).
  • Evaluation Benchmarks:
    • LLaVA-Bench (MSA): An Arabic version of the LLaVA-Bench, used for evaluating performance in Modern Standard Arabic.
  • Dallah-Bench (Dialects): A newly introduced benchmark specifically designed to assess dialectal responses in six Arabic dialects, using a subset of the Henna dataset.

Key Insights and Ideas

  • The Power of Dialect-Specific Data: Fine-tuning on carefully curated dialectal data is crucial for achieving authentic and accurate responses in MLLMs for diverse linguistic communities.
  • Translate-and-Filter is Key: A robust data preprocessing pipeline, especially one that includes back-translation and similarity filtering, is essential for creating high-quality multimodal datasets in low-resource languages.
  • LLaVA Architecture's Adaptability: The LLaVA framework proves to be a flexible foundation for building specialized MLLMs, capable of adapting to different linguistic requirements with appropriate LLM integration.
  • Dialectal Nuances Matter: Even subtle differences in dialect can significantly impact model performance, highlighting the need for models that can distinguish and respond appropriately to these variations.

Key Results

Dallah demonstrates state-of-the-art performance on Arabic MLLM tasks.

  • LLaVA-Bench (MSA): Dallah achieved an average score of 89.74%, outperforming baseline models like Peacock (69.35%) and PALO (78.07%) across various evaluation metrics (Conversations, Details, Complex Reasoning) and evaluators (GPT-4, Command R+, GPT-4-Turbo, Human).
  • Dallah-Bench (Dialects): Human evaluations showed Dallah performing well in dialect authenticity and content accuracy, with higher ratings from Cohere Command R+ compared to GPT-4-Turbo, suggesting better alignment with human judgment for dialectal tasks.
  • Qualitative analysis showed Dallah's ability to generate detailed descriptions, provide nutritional advice, and handle dialectal variations effectively, though it sometimes struggled with recognizing Arabic figures and was prone to hallucinations.

Limitations and Future Work

The authors acknowledge several limitations:

  • Representation of Arabic Culture: The model struggles with recognizing Arabic figures, indicating a need for more diverse cultural datasets.
  • Hallucination Control: Like many LLMs, Dallah can generate inaccurate information, requiring advanced techniques and more robust datasets.
  • Dialect Variation and Mixing: The model sometimes mixes similar dialects and struggles with dialects close to MSA, suggesting a need for more extensive data collection and fine-tuning.
  • Arabic Text Recognition in Images: Dallah cannot effectively recognize Arabic text within images due to a lack of annotated datasets.
  • Energy Efficiency: Training large MLLMs like Dallah is computationally intensive and not energy-efficient.

Future work could focus on expanding dialect coverage, improving hallucination control, developing Arabic OCR capabilities for multimodal models, and exploring more sophisticated evaluation metrics that better capture cultural and linguistic subtleties.

Summary updated

BibTeX
@inproceedings{alwajih2024dallah,
  author = {Fakhraddin Alwajih and Gagan Bhatia and Muhammad Abdul-Mageed},
  title = {{Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure