Abstract
Arabic is a linguistically complex macro-language characterized by diglossia—the sharp divide between Modern Standard Arabic (MSA) and regional dialects. Existing embedding models often treat Arabic as a monolith, failing to capture these nuances. This paper introduces Swan, a family of state-of-the-art Arabic-centric embedding models, and ArabicMTEB, a comprehensive benchmark comprising 94 datasets across 8 task clusters. By utilizing instruction tuning and synthetic data augmentation, Swan outperforms leading multilingual models like Multilingual-E5, particularly in dialectal and cultural contexts, while offering significant deployment efficiency.
Introduction: Addressing the Diglossia Gap
The primary challenge in Arabic Natural Language Processing (NLP) is the coexistence of formal MSA and diverse regional dialects (e.g., Egyptian, Moroccan, Levantine). Most current text embedding models are optimized for English or Chinese and struggle with the morphological richness and regional variations of Arabic. Furthermore, the lack of a unified, large-scale benchmark has hindered the evaluation of models on their ability to handle the specific linguistic and cultural nuances of the Arab world.
The ArabicMTEB Benchmark
To provide a rigorous evaluation framework, the researchers developed ArabicMTEB. This benchmark is the most extensive to date for Arabic, featuring 94 datasets categorized into eight distinct task clusters.
Beyond standard tasks, ArabicMTEB includes specialized subsets:
- Dialectal ArabicMTEB: 19 datasets focused on regional varieties.
- Cultural ArabicMTEB: A novel subset evaluating country-level cultural knowledge across 20 Arab nations.
- Domain-Specific Subsets: Covering News, Finance, Legal, and Medical fields.
The Swan Model Family
The authors developed two versions of the Swan model to balance performance and efficiency:
- Swan-Small: Based on ARBERTv2 (164M parameters), optimized for edge deployment.
- Swan-Large: Based on ArMistral-7B (7.2B parameters), utilizing Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning.
Key Methodological Innovations
Swan models incorporate Instruction Tuning, allowing them to follow task-specific prompts (e.g., "Retrieve parallel sentences"). This guides the model to represent text more effectively for specific goals. Additionally, the team addressed the scarcity of dialectal data through Synthetic Data Augmentation. Using the Command R+ model, they generated high-quality, paragraph-based synthetic data for both MSA and specific dialects.
Results and Performance
Swan models achieved state-of-the-art results across the ArabicMTEB benchmark:
- Overall Performance: Swan-Large reached an average score of 62.45, surpassing Multilingual-E5-large (61.65). Swan-Small (57.33) also outperformed its base-sized competitors.
- Dialectal Dominance: In dialect-specific tasks, Swan-Large achieved a score of 70.45, demonstrating the effectiveness of including dialectal data in training.
- Cultural Awareness: The Cultural ArabicMTEB subset showed that Swan can distinguish between specific regional traditions, a task where generic models often fail.
- Efficiency: Swan-Large is estimated to be approximately 13 times more cost-effective to deploy for large-scale tasks compared to proprietary models like OpenAI’s
text-embedding-3-large.
Conclusion
Swan and ArabicMTEB represent a significant step forward in making Arabic NLP more inclusive of regional and cultural diversity. By moving away from a monolithic view of the language and leveraging synthetic data, the researchers have provided a powerful toolset for developers building Arabic-centric AI applications. Future work will focus on expanding dialectal coverage and exploring multimodal embeddings.
Summary updated