Abstract
ALLaM is a new family of large language models, ranging from 7B to 70B parameters, specifically optimized for the Arabic and English languages. Developed by the National Center for AI (SDAIA), ALLaM utilizes a "Second Language Acquisition" approach that combines vocabulary expansion, mixed-language pretraining, and high-quality human alignment. The resulting models achieve state-of-the-art results on Arabic benchmarks while simultaneously enhancing performance on English tasks.
Introduction
Despite the rapid advancement of Large Language Models (LLMs), frontier models like GPT-4 and Llama-3 remain predominantly English-centric. This creates a significant gap in linguistic nuance and cultural representation for the world's 400 million Arabic speakers. Training high-quality models from scratch for every language is computationally expensive and environmentally taxing.
To address this, the researchers at the National Center for AI (SDAIA) focused on two primary challenges: efficient adaptation of existing strong English models to Arabic without "catastrophic forgetting," and cultural alignment to ensure the model respects regional norms and history rather than merely providing literal translations of Western concepts.
Methodology
Second Language Acquisition (SLA)
Instead of initializing weights randomly, the ALLaM series (7B, 13B, and 70B) was built using Llama-2 weights as a foundation. This allowed the models to leverage existing reasoning capabilities while focusing new training cycles on linguistic adaptation.
Vocabulary Expansion and Embedding Initialization
Standard English tokenizers process Arabic inefficiently, often breaking words into character-level chunks. To resolve this, the team merged the Llama-2 tokenizer with a specialized Arabic tokenizer. This significantly reduced the "fertility rate"—the number of tokens required per word—making the model faster and more context-efficient for Arabic text.
To accelerate learning, new Arabic token embeddings were initialized by averaging the representations of existing tokens that formed the same sub-words, rather than using random initialization.
Alignment and Fine-Tuning
The models underwent a two-stage alignment process:
- Supervised Fine-Tuning (SFT): Utilizing the "Ultra-Instinct" dataset (refined to 6M high-quality samples), the model's instruction-following capabilities were honed.
- Direct Preference Optimization (DPO): 25,000 human-verified samples were used to align outputs with human preferences, emphasizing safety and cultural relevance.
Datasets and Training
The pretraining phase involved 540B Arabic tokens, balanced between natural Arabic (web crawls, books, news) and machine-translated English data. This was mixed with 4T English tokens.
A key finding was that machine-translated data stabilized training by reducing gradient spikes and acting as a linguistic bridge. Ablation studies determined that a 45/55 Arabic-to-English ratio was optimal for acquiring new language skills while boosting original English proficiency.
Key Results
ALLaM-70B currently stands as a top-tier model for the Arabic language, achieving state-of-the-art performance on five out of eight major benchmarks, including MMLU Arabic and the ETEC (Saudi school exam) set.
As shown in the evaluation metrics, ALLaM-70B frequently outperforms much newer models like Llama-3-70B in Arabic tasks. Remarkably, the continued pretraining on Arabic did not degrade English performance; instead, the aligned ALLaM models often scored higher on English MMLU than the base Llama-2 models they were derived from. In human preference evaluations, ALLaM-13B maintained a higher win rate against larger competitors like Command-R+.
Limitations and Future Work
While ALLaM represents a significant leap for Arabic LLMs, the authors acknowledge ongoing challenges. Web-scale training data can still harbor toxic language or societal biases. Furthermore, multiple-choice benchmarks are susceptible to data leakage, necessitating more robust, human-centric evaluation frameworks. Future research will explore Proximal Policy Optimization (PPO) for finer alignment and further refine data curation to reduce hallucinations in regional contexts.
Summary updated