3LM: Bridging Arabic, STEM, and Code through Benchmarking

Basma El Amel Boussaha, Leen Al Qadi, Mugariya Farooq, Shaikha Alsuwaidi, Giulia Campesan, Ahmed Alzubaidi, Mohammed Alyafeai, Hakim Hacid2025 · DOI

Abstract: Basma El Amel Boussaha, Leen Al Qadi, Mugariya Farooq, Shaikha Alsuwaidi, Giulia Campesan, Ahmed Alzubaidi, Mohammed Alyafeai, Hakim Hacid. Proceedings of The Third Arabic Natural Language Processing Conference.

2025.

Introduction: The Scientific Gap in Arabic LLM Evaluation

As the ecosystem for Arabic Large Language Models (LLMs) matures, the focus of evaluation must shift from general linguistic fluency to specialized technical domains. Current benchmarks often emphasize "soft" dimensions—dialectal variations, cultural nuances, or general language understanding. However, a significant "scientific gap" remains: existing STEM benchmarks for Arabic are frequently closed-source or suffer from poor translation quality.

The 3LM (Arabic, STEM, and Code) framework addresses this by introducing a high-quality, scientifically grounded evaluation suite. It moves beyond simple pattern matching to test whether models can truly reason in Arabic across physics, chemistry, biology, mathematics, and programming.

Methodology: A Multi-Pronged Curation Pipeline

To ensure technical accuracy and linguistic naturalness, the researchers developed a sophisticated pipeline for data acquisition and quality control.

Overview of the 3LM data collection and curation pipeline, including OCR, translation, and human verification stages.
  1. Native STEM Curation: Arabic textbooks and worksheets were processed using Pix2Tex, a specialized math-based OCR pipeline, to preserve LaTeX formatting. This was followed by LLM-based extraction and rigorous manual human verification.
  2. Synthetic Generation: Utilizing the YourBench framework, the team generated multi-hop questions from scientific texts. These questions require synthesizing information across documents, moving beyond simple fact retrieval.
  3. Code Translation: The industry-standard HumanEval+ and MBPP+ benchmarks were adapted into Arabic. To prevent "translationese," a back-translation pipeline was implemented, using ROUGE-L scores and manual developer reviews to ensure semantic integrity.
  4. Robustness Testing: The suite includes "Robustness under Distractor Perturbation" (RDP) analysis, where correct answers are removed or "None of the above" is added to detect superficial pattern-matching behaviors.

The 3LM Suite: Composition and Scope

The resulting dataset provides a comprehensive look at model capabilities:

  • Native STEM: 865 QA pairs sourced from educational materials in Biology, Chemistry, Geography, Math, and Physics.
  • Synthetic STEM: 1,744 automatically generated, high-difficulty questions.
  • Code (EvalPlus-Ar): 542 high-quality translated prompts from HumanEval and MBPP, including expanded test suites.

Analysis and Key Findings

The evaluation of over 40 models—including the Gemma, Qwen, and Llama families, alongside Arabic-centric models like Jais—revealed several critical insights.

The Difficulty of Synthetic Reasoning

One of the most significant findings was the disparity in difficulty between native and synthetic sets. While native questions followed a standard Gaussian distribution, synthetic questions were consistently more challenging, providing a better stress test for state-of-the-art models.

Comparison of question difficulty distribution between native and synthetic benchmarks, highlighting the increased complexity of synthetic tasks.

Reasoning vs. Pattern Matching

The RDP tests exposed a widespread "guessing" problem. Performance dropped significantly across all models when distractors were introduced, indicating that many LLMs rely on superficial cues rather than deep reasoning. Notably, instruct-tuned models showed higher resilience than base models, underscoring the importance of supervised fine-tuning for reasoning tasks.

Language-Agnostic Logic in Code

The study found a near-perfect correlation (r = 0.97) between performance on English and Arabic code benchmarks. This suggests that programming logic is largely language-independent; once a model masters the logic, the prompt language becomes secondary, provided the translation is accurate.

Performance Leaders

  • Top Performers: Gemma-3-27B and Qwen2.5-72B led the rankings, with Qwen2.5-72B-Instruct achieving 93.06% on Native MCQs.
  • Coding Excellence: Qwen3-14B-Base outperformed many larger models in code generation tasks.
  • Arabic-Centric Models: While strong in general NLP, models like Jais and AceGPT still trail multilingual giants in specialized STEM and Code reasoning.

Conclusion and Future Directions

While 3LM sets a new standard for Arabic evaluation, the authors identify key areas for growth. The current suite focuses on K-12 levels and is text-only. Future iterations should incorporate university-level content and multimodal elements—such as diagrams, chemical structures, and charts—which are essential for complete scientific reasoning. By providing an open-source "North Star," 3LM empowers the community to build more capable and scientifically literate Arabic AI.

Summary updated

BibTeX
@misc{boussaha2025lm,
  author = {Basma El Amel Boussaha and Leen Al Qadi and Mugariya Farooq and Shaikha Alsuwaidi and Giulia Campesan and Ahmed Alzubaidi and Mohammed Alyafeai and Hakim Hacid},
  title = {{3LM: Bridging Arabic, STEM, and Code through Benchmarking}},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.4},
}
Figure