NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities

Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, Muhammad Abdul-MageedEMNLP 2025

TL;DR

This paper introduces NileChat, a novel framework and resulting LLM designed to bridge the gap between general-purpose Large Language Models (LLMs) and the specific linguistic and cultural needs of local communities, particularly those speaking low-resource Arabic dialects. By developing a multi-stage data augmentation pipeline that incorporates cultural heritage and values, NileChat significantly enhances performance on understanding, translation, and cultural alignment tasks, outperforming existing models.

What Problem Does This Paper Address?

Large Language Models (LLMs) have achieved remarkable success, but their capabilities are unevenly distributed across languages, with a significant deficit in low-resource languages and dialects. This is largely due to pre-training corpora being dominated by high-resource languages, leading to models that are culturally aligned with their source languages. This "cultural encapsulation" results in LLMs that fail to represent the heritage and values of local communities, leading to misunderstandings and a lack of genuine interaction. Current adaptation techniques often focus solely on linguistic deficits, neglecting the crucial cultural dimension. This paper addresses the critical need to create LLMs that are not only linguistically proficient but also culturally aware and aligned with the specific values of diverse, under-represented communities.

Methods

The authors propose a novel data augmentation pipeline for continued pre-training, focusing on three key dimensions: Language, Cultural Heritage, and Cultural Values (LHV). The pipeline involves:

  1. Machine Translation (MT): Translating high-quality English educational content into the target low-resource language (Egyptian and Moroccan Arabic) using a specialized teacher LLM. This aims to improve linguistic fluency and coherence.
  2. Controlled Synthetic Data Generation: Creating diverse texts (stories, conversations, etc.) that reflect specific cultural heritage concepts and local values. This is achieved by simulating scenarios with local personas, defined by demographic characteristics and moral values derived from the World Values Survey (WVS). Local context from news articles and cultural elements from Wikipedia are also incorporated.
  3. Retrieval for Local Cultural Heritage: Querying a search engine with a list of cultural concepts to retrieve and parse culturally specific web content, further enriching the training data.

This multi-stage process enables controlled text generation and retrieval-augmented pre-training, ensuring cultural and value alignment. The resulting augmented dataset is then used to continue pre-training a base LLM (Qwen-2.5-3B), followed by supervised fine-tuning (SFT).

Datasets

  • Training Data:
    • Machine Translation Data: 5.5 million texts from the Fineweb-edu dataset translated into Egyptian and Moroccan Arabic.
  • Controlled Synthetic Generation Data: Includes persona descriptions (1,200 based on WVS), local news context (approx. 1.5M Egyptian and 800K Moroccan articles), cultural heritage concepts (from Wikipedia), dialectal glosses, expressions, and utterances, and augmented data from Egyptian and Moroccan television shows.
  • Retrieval Data: Cultural concepts from Morocco (6,500) and Egypt (4,500) queried via Brave Search API.
  • Existing Publicly Available Data: For Egyptian and Moroccan dialects, MSA, English, French, Math, and Code.
  • Supervised Fine-Tuning (SFT) Data: A comprehensive set constructed from:
  • Translation of the SmolTalk dataset into MOR, EGY, French, and MSA.
  • Synthetic generation of dialectal Q&A pairs using retrieved cultural heritage data.
  • Incorporation of the Darija-SFT-Mixture MOR dataset.
  • Translation of the TULU-V2-mix dataset into EGY.
  • Augmented understanding and generation tasks from ORCA and Dolphin benchmarks.
  • Evaluation Benchmarks:
    • Understanding: MMLU, HellaSwag, and Belebele (adapted for EGY and MOR dialects).
  • Cultural Knowledge: Palm benchmark (Egypt and Morocco specific test set).
  • Translation: Flores-200 dataset and an in-house, human-curated dataset of authentic EGY and MOR utterances.
  • Value Alignment: Adapted WVS questions using the Social Value Alignment (SVA) metric.

The paper also highlights the public release of new datasets for Egyptian and Moroccan Arabic dialects.

Key Insights and Ideas

  • Cultural Nuance is Crucial: Linguistic fluency alone is insufficient for genuine cultural alignment. LLMs need to be imbued with specific cultural heritage and values to truly serve local communities.
  • Multi-Stage Data Augmentation is Effective: Combining machine translation, controlled synthetic generation (with personas and local context), and retrieval is a powerful strategy for creating culturally rich pre-training data for low-resource languages.
  • Personas Drive Cultural Alignment: Using detailed persona descriptions that capture moral values and demographic attributes is key to generating texts that reflect community-specific worldviews.
  • Dialectal Nuances Matter: The paper emphasizes the importance of capturing dialectal variations, including Arabizi (Arabic script transliterated into Latin characters), for authentic communication.
  • Teacher LLMs are Valuable Tools: A proficient teacher LLM can be leveraged to translate and generate diverse, culturally relevant content, overcoming the scarcity of high-quality data for low-resource languages.

Key Results

  • NileChat Achieves State-of-the-Art (SoTA): NileChat (3B parameters) demonstrates SoTA performance on MMLU, HellaSwag, and Belebele benchmarks for both Egyptian and Moroccan dialects, significantly outperforming its baseline (Qwen2.5-3B-instruct) and larger Arabic-focused models like AceGPT-13B and Jais-13B.
  • Enhanced Cultural Knowledge: NileChat achieves significantly higher scores on the Palm benchmark for cultural knowledge (5.72 for EGY, 5.86 for MOR) compared to its baseline (2.86 for EGY, 2.31 for MOR).
  • Strong Translation Performance: NileChat achieves the highest average translation quality (spBLEU: 21.32) across evaluated models, outperforming even larger competitors on its in-house, human-curated dataset.
  • Substantial Value Alignment: NileChat shows substantial improvements over baselines in value alignment evaluation across most societal dimensions for both Moroccan and Egyptian contexts, demonstrating successful steering towards culturally aligned positions.
  • Efficiency: The proposed method enables adaptation without requiring pre-training LLMs from scratch, leveraging existing models.

Limitations and Future Work

  • Teacher Model Dependency for Low-Resource Languages: The method relies on a teacher model proficient in generating low-resource languages, which might not be available for extremely under-resourced languages.
  • Supervised Fine-Tuning Data: The reliance on translated data for SFT might impact nuanced performance in target languages and cultures.
  • Susceptibility to Hallucination: As a 3B parameter model, NileChat is relatively small and may be more prone to hallucination compared to larger architectures.
  • Computational Cost of Synthetic Data Generation: Generating synthetic data is computationally intensive, especially for long documents and extensive input contexts.
  • Absence of Explicit Safety Alignment: The model has not undergone dedicated safety tuning, which is acknowledged as important future work.
  • Limited Generation of Subtle Details: While capturing cultural nuances, the generated texts may still lack the very subtle implicit knowledge, humor, or sarcasm present in real-world communication.

Future work could explore adapting the methodology to extremely under-resourced languages, developing more robust safety alignment procedures, and further investigating the generation of subtle cultural details.

Summary updated

BibTeX
@inproceedings{mekki2025nilechat,
  author = {Abdellah El Mekki and Houdaifa Atou and Omer Nacar and Shady Shehata and Muhammad Abdul-Mageed},
  title = {{NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure