ALLaM: Large Language Models for Arabic and English

M. Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan Alrashed, Faisal Mirza, Shaykhah Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, Haidar Khan

Abstract

ALLaM is a new family of large language models, ranging from 7B to 70B parameters, specifically optimized for the Arabic and English languages. Developed by the National Center for AI (SDAIA), ALLaM utilizes a "Second Language Acquisition" approach that combines vocabulary expansion, mixed-language pretraining, and high-quality human alignment. The resulting models achieve state-of-the-art results on Arabic benchmarks while simultaneously enhancing performance on English tasks.

Introduction

Despite the rapid advancement of Large Language Models (LLMs), frontier models like GPT-4 and Llama-3 remain predominantly English-centric. This creates a significant gap in linguistic nuance and cultural representation for the world's 400 million Arabic speakers. Training high-quality models from scratch for every language is computationally expensive and environmentally taxing.

To address this, the researchers at the National Center for AI (SDAIA) focused on two primary challenges: efficient adaptation of existing strong English models to Arabic without "catastrophic forgetting," and cultural alignment to ensure the model respects regional norms and history rather than merely providing literal translations of Western concepts.

Methodology

Second Language Acquisition (SLA)

Instead of initializing weights randomly, the ALLaM series (7B, 13B, and 70B) was built using Llama-2 weights as a foundation. This allowed the models to leverage existing reasoning capabilities while focusing new training cycles on linguistic adaptation.

Vocabulary Expansion and Embedding Initialization

Standard English tokenizers process Arabic inefficiently, often breaking words into character-level chunks. To resolve this, the team merged the Llama-2 tokenizer with a specialized Arabic tokenizer. This significantly reduced the "fertility rate"—the number of tokens required per word—making the model faster and more context-efficient for Arabic text.

Fertility rate comparison across models, showing ALLaM's improved efficiency in processing Arabic text compared to the Llama-2 baseline.

To accelerate learning, new Arabic token embeddings were initialized by averaging the representations of existing tokens that formed the same sub-words, rather than using random initialization.

Alignment and Fine-Tuning

The models underwent a two-stage alignment process:

  1. Supervised Fine-Tuning (SFT): Utilizing the "Ultra-Instinct" dataset (refined to 6M high-quality samples), the model's instruction-following capabilities were honed.
  2. Direct Preference Optimization (DPO): 25,000 human-verified samples were used to align outputs with human preferences, emphasizing safety and cultural relevance.

Datasets and Training

The pretraining phase involved 540B Arabic tokens, balanced between natural Arabic (web crawls, books, news) and machine-translated English data. This was mixed with 4T English tokens.

A key finding was that machine-translated data stabilized training by reducing gradient spikes and acting as a linguistic bridge. Ablation studies determined that a 45/55 Arabic-to-English ratio was optimal for acquiring new language skills while boosting original English proficiency.

Key Results

ALLaM-70B currently stands as a top-tier model for the Arabic language, achieving state-of-the-art performance on five out of eight major benchmarks, including MMLU Arabic and the ETEC (Saudi school exam) set.

Performance of ALLaM Instruct models compared to leading open-source models on Arabic and English MMLU benchmarks.

As shown in the evaluation metrics, ALLaM-70B frequently outperforms much newer models like Llama-3-70B in Arabic tasks. Remarkably, the continued pretraining on Arabic did not degrade English performance; instead, the aligned ALLaM models often scored higher on English MMLU than the base Llama-2 models they were derived from. In human preference evaluations, ALLaM-13B maintained a higher win rate against larger competitors like Command-R+.

Limitations and Future Work

While ALLaM represents a significant leap for Arabic LLMs, the authors acknowledge ongoing challenges. Web-scale training data can still harbor toxic language or societal biases. Furthermore, multiple-choice benchmarks are susceptible to data leakage, necessitating more robust, human-centric evaluation frameworks. Future research will explore Proximal Policy Optimization (PPO) for finer alignment and further refine data curation to reduce hallucinations in regional contexts.

Summary updated

Abstract

We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models.
BibTeX
@inproceedings{bari2025allam,
  author = {M. Saiful Bari and Yazeed Alnumay and Norah A. Alzahrani and Nouf M. Alotaibi and Hisham Abdullah Alyahya and Sultan Alrashed and Faisal Mirza and Shaykhah Alsubaie and Hassan A. Alahmed and Ghadah Alabduljabbar and Raghad Alkhathran and Yousef Almushayqih and Raneem Alnajim and Salman Alsubaihi and Maryam Al Mansour and Majed Alrubaian and Ali Alammari and Zaki Alawami and Abdulmohsen Al-Thubaity and Ahmed Abdelali and Jeril Kuriakose and Abdalghani Abujabal and Nora Al-Twairesh and Areeb Alowisheq and Haidar Khan},
  title = {{ALLaM: Large Language Models for Arabic and English}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=MscdsFVZrN},
}
Figure