Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarks

Gagan Bhatia, El Moatez Billah Nagoudi, Abdellah El Mekki, Fakhraddin Alwajih, Muhammad Abdul-MageedNAACL 2025 · DOI

Abstract

Arabic is a linguistically complex macro-language characterized by diglossia—the sharp divide between Modern Standard Arabic (MSA) and regional dialects. Existing embedding models often treat Arabic as a monolith, failing to capture these nuances. This paper introduces Swan, a family of state-of-the-art Arabic-centric embedding models, and ArabicMTEB, a comprehensive benchmark comprising 94 datasets across 8 task clusters. By utilizing instruction tuning and synthetic data augmentation, Swan outperforms leading multilingual models like Multilingual-E5, particularly in dialectal and cultural contexts, while offering significant deployment efficiency.

Introduction: Addressing the Diglossia Gap

The primary challenge in Arabic Natural Language Processing (NLP) is the coexistence of formal MSA and diverse regional dialects (e.g., Egyptian, Moroccan, Levantine). Most current text embedding models are optimized for English or Chinese and struggle with the morphological richness and regional variations of Arabic. Furthermore, the lack of a unified, large-scale benchmark has hindered the evaluation of models on their ability to handle the specific linguistic and cultural nuances of the Arab world.

The ArabicMTEB Benchmark

To provide a rigorous evaluation framework, the researchers developed ArabicMTEB. This benchmark is the most extensive to date for Arabic, featuring 94 datasets categorized into eight distinct task clusters.

The eight task clusters included in the ArabicMTEB benchmark, ranging from clustering and retrieval to bitext mining.

Beyond standard tasks, ArabicMTEB includes specialized subsets:

  • Dialectal ArabicMTEB: 19 datasets focused on regional varieties.
  • Cultural ArabicMTEB: A novel subset evaluating country-level cultural knowledge across 20 Arab nations.
  • Domain-Specific Subsets: Covering News, Finance, Legal, and Medical fields.

The Swan Model Family

The authors developed two versions of the Swan model to balance performance and efficiency:

  • Swan-Small: Based on ARBERTv2 (164M parameters), optimized for edge deployment.
  • Swan-Large: Based on ArMistral-7B (7.2B parameters), utilizing Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning.

Key Methodological Innovations

Swan models incorporate Instruction Tuning, allowing them to follow task-specific prompts (e.g., "Retrieve parallel sentences"). This guides the model to represent text more effectively for specific goals. Additionally, the team addressed the scarcity of dialectal data through Synthetic Data Augmentation. Using the Command R+ model, they generated high-quality, paragraph-based synthetic data for both MSA and specific dialects.

An example of the synthetic data generation process, showing a generated Arabic retrieval task with a query, a positive document, and a hard negative.

Results and Performance

Swan models achieved state-of-the-art results across the ArabicMTEB benchmark:

  • Overall Performance: Swan-Large reached an average score of 62.45, surpassing Multilingual-E5-large (61.65). Swan-Small (57.33) also outperformed its base-sized competitors.
  • Dialectal Dominance: In dialect-specific tasks, Swan-Large achieved a score of 70.45, demonstrating the effectiveness of including dialectal data in training.
  • Cultural Awareness: The Cultural ArabicMTEB subset showed that Swan can distinguish between specific regional traditions, a task where generic models often fail.
  • Efficiency: Swan-Large is estimated to be approximately 13 times more cost-effective to deploy for large-scale tasks compared to proprietary models like OpenAI’s text-embedding-3-large.

Conclusion

Swan and ArabicMTEB represent a significant step forward in making Arabic NLP more inclusive of regional and cultural diversity. By moving away from a monolithic view of the language and leveraging synthetic data, the researchers have provided a powerful toolset for developers building Arabic-centric AI applications. Future work will focus on expanding dialectal coverage and exploring multimodal embeddings.

Summary updated

BibTeX
@inproceedings{bhatia2025swan,
  author = {Gagan Bhatia and El Moatez Billah Nagoudi and Abdellah El Mekki and Fakhraddin Alwajih and Muhammad Abdul-Mageed},
  title = {{Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.263},
}
Figure