Abstract: Basma El Amel Boussaha, Leen Al Qadi, Mugariya Farooq, Shaikha Alsuwaidi, Giulia Campesan, Ahmed Alzubaidi, Mohammed Alyafeai, Hakim Hacid. Proceedings of The Third Arabic Natural Language Processing Conference.
2025.
Introduction: The Scientific Gap in Arabic LLM Evaluation
As the ecosystem for Arabic Large Language Models (LLMs) matures, the focus of evaluation must shift from general linguistic fluency to specialized technical domains. Current benchmarks often emphasize "soft" dimensions—dialectal variations, cultural nuances, or general language understanding. However, a significant "scientific gap" remains: existing STEM benchmarks for Arabic are frequently closed-source or suffer from poor translation quality.
The 3LM (Arabic, STEM, and Code) framework addresses this by introducing a high-quality, scientifically grounded evaluation suite. It moves beyond simple pattern matching to test whether models can truly reason in Arabic across physics, chemistry, biology, mathematics, and programming.
Methodology: A Multi-Pronged Curation Pipeline
To ensure technical accuracy and linguistic naturalness, the researchers developed a sophisticated pipeline for data acquisition and quality control.
- Native STEM Curation: Arabic textbooks and worksheets were processed using Pix2Tex, a specialized math-based OCR pipeline, to preserve LaTeX formatting. This was followed by LLM-based extraction and rigorous manual human verification.
- Synthetic Generation: Utilizing the YourBench framework, the team generated multi-hop questions from scientific texts. These questions require synthesizing information across documents, moving beyond simple fact retrieval.
- Code Translation: The industry-standard HumanEval+ and MBPP+ benchmarks were adapted into Arabic. To prevent "translationese," a back-translation pipeline was implemented, using ROUGE-L scores and manual developer reviews to ensure semantic integrity.
- Robustness Testing: The suite includes "Robustness under Distractor Perturbation" (RDP) analysis, where correct answers are removed or "None of the above" is added to detect superficial pattern-matching behaviors.
The 3LM Suite: Composition and Scope
The resulting dataset provides a comprehensive look at model capabilities:
- Native STEM: 865 QA pairs sourced from educational materials in Biology, Chemistry, Geography, Math, and Physics.
- Synthetic STEM: 1,744 automatically generated, high-difficulty questions.
- Code (EvalPlus-Ar): 542 high-quality translated prompts from HumanEval and MBPP, including expanded test suites.
Analysis and Key Findings
The evaluation of over 40 models—including the Gemma, Qwen, and Llama families, alongside Arabic-centric models like Jais—revealed several critical insights.
The Difficulty of Synthetic Reasoning
One of the most significant findings was the disparity in difficulty between native and synthetic sets. While native questions followed a standard Gaussian distribution, synthetic questions were consistently more challenging, providing a better stress test for state-of-the-art models.
Reasoning vs. Pattern Matching
The RDP tests exposed a widespread "guessing" problem. Performance dropped significantly across all models when distractors were introduced, indicating that many LLMs rely on superficial cues rather than deep reasoning. Notably, instruct-tuned models showed higher resilience than base models, underscoring the importance of supervised fine-tuning for reasoning tasks.
Language-Agnostic Logic in Code
The study found a near-perfect correlation (r = 0.97) between performance on English and Arabic code benchmarks. This suggests that programming logic is largely language-independent; once a model masters the logic, the prompt language becomes secondary, provided the translation is accurate.
Performance Leaders
- Top Performers: Gemma-3-27B and Qwen2.5-72B led the rankings, with Qwen2.5-72B-Instruct achieving 93.06% on Native MCQs.
- Coding Excellence: Qwen3-14B-Base outperformed many larger models in code generation tasks.
- Arabic-Centric Models: While strong in general NLP, models like Jais and AceGPT still trail multilingual giants in specialized STEM and Code reasoning.
Conclusion and Future Directions
While 3LM sets a new standard for Arabic evaluation, the authors identify key areas for growth. The current suite focuses on K-12 levels and is text-only. Future iterations should incorporate university-level content and multimodal elements—such as diagrams, chemical structures, and charts—which are essential for complete scientific reasoning. By providing an open-source "North Star," 3LM empowers the community to build more capable and scientifically literate Arabic AI.
Summary updated