TL;DR
This paper introduces 3LM, a novel benchmark suite designed to evaluate Large Language Models (LLMs) on Arabic STEM and code generation tasks. By creating native and synthetically generated STEM question-answer pairs, and translating established code benchmarks into Arabic, 3LM addresses a critical gap in evaluating LLMs for the Arabic language in these crucial domains.
What Problem Does This Paper Address?
The rapid advancement of LLMs has highlighted the need for robust evaluation benchmarks. While many benchmarks exist for Arabic, they often focus on linguistic, cultural, or religious content. There's a significant gap in evaluating LLMs for Arabic in STEM (Science, Technology, Engineering, and Mathematics) and code generation, domains that are increasingly important for real-world applications. Existing Arabic STEM benchmarks are either limited in scope or not publicly accessible, hindering progress in this area. This paper aims to fill this void by providing a comprehensive, open-source evaluation suite for Arabic LLMs in these underrepresented fields.
Methods
The creation of the 3LM benchmark suite involved a multi-stage process:
- Data Collection: STEM content was systematically gathered from online educational resources, including websites, open question banks, worksheets, exams, and textbooks. PDFs were filtered for relevance and educational level, prioritizing middle and high school content. A specialized OCR pipeline, including a Math-based OCR model (Pix2Tex), was used to extract text and convert mathematical notation into LaTeX.
- Native Benchmark Creation: Multiple-choice questions (MCQs) were extracted from the collected native Arabic educational content. A four-stage pipeline, leveraging the Qwen3-235B-A22B model, was used for question-answer extraction, classification (type, difficulty, domain, visual dependency), and format standardization.
- Synthetic Benchmark Creation: The YourBench framework was adapted for Arabic to synthetically generate STEM questions. This involved an LLM-powered pipeline for ingestion, summarization, chunking, question generation, and analysis. Rigorous filtering ensured quality and diversity.
- Code Benchmark Creation: Two widely used English code benchmarks, HumanEval+ and MBPP+, were translated into Arabic using GPT-40. A rigorous machine translation pipeline with backtranslation and human-in-the-loop validation was employed to ensure high-quality and faithful translations. This resulted in HumanEval-Arabic and MBPP-Arabic, forming the EvalPlus-Arabic suite.
Datasets
The paper introduces the 3LM benchmark suite, comprising three distinct datasets:
- Native STEM QA: A collection of native Arabic multiple-choice questions sourced directly from educational materials.
- Synthetic STEM QA: Automatically generated STEM questions based on Arabic textbooks and educational platforms, using the YourBench framework.
- EvalPlus-Arabic: A suite of translated code generation benchmarks, including HumanEval-Arabic and MBPP-Arabic, adapted from the English HumanEval+ and MBPP+ datasets.
Key Insights and Ideas
- Bridging the Domain Gap: The most significant contribution is the creation of native Arabic benchmarks for STEM and code, addressing a critical underrepresentation in current LLM evaluation.
- Quality through Rigor: The paper emphasizes a meticulous curation process, including human-in-the-loop validation for code translation and systematic classification for STEM questions, to ensure high-quality and reliable benchmarks.
- Model Behavior Insights: The evaluation of various LLMs on 3LM reveals interesting patterns, such as larger models sometimes underperforming smaller ones in completion tasks, and highlights the importance of instruction tuning for robustness.
- Code Generalization: The strong correlation between English and Arabic code generation scores suggests that LLMs can generalize code capabilities across languages when prompts are translated, but this doesn't always translate to Arabic NLP performance.
Key Results
- The 3LM benchmark suite contains 865 native QA pairs, 1,744 synthetic questions, and 542 translated code prompts.
- Models generally perform better on MCQ formats compared to completion formats across all scales.
- For base models, Gemma3-27B and Qwen3-30B-A3B showed top performance across STEM domains, with Gemma3-27B achieving the highest overall average.
- For instruct models, Qwen2.5-72B-Instruct and Qwen3-32B achieved the highest scores on the Arabic code suite.
- A consistent performance drop was observed under distractor perturbation, with base models showing larger declines than instruct-tuned ones. Large instruct models like Qwen2.5-72B and Llama-3.3-70B remained relatively stable.
- The correlation between Arabic NLP and code scores weakens considerably for instruct models compared to base models, suggesting specialization.
Limitations and Future Work
- Limited Scope: The benchmark primarily targets middle and high school content, potentially limiting assessment of advanced university-level scientific concepts.
- Synthetic Bias: The synthetic benchmark generation process may inherit biases from the underlying language models, influencing question difficulty and topic coverage.
- Mixed-Language Code: The translated code prompts retain English variable names and function signatures, which may not fully capture challenges of purely Arabic code contexts.
- Text-Based Only: The benchmark excludes visual elements like diagrams and graphs, which are crucial for STEM understanding.
- Future Work: The authors suggest expanding the benchmark to include more advanced topics and visual elements. Further research could explore the impact of purely Arabic code contexts and investigate the nuances of LLM reasoning in Arabic STEM domains.
Summary updated