Abstract: Benchmarking Large Language Models (LLMs) for low-resource languages remains a significant challenge due to the scarcity of high-quality, standardized datasets. This study evaluates 53 LLMs on Luxembourgish using official human language proficiency exams (A1–C2) from the Institut National des Langues Luxembourg (INLL). We demonstrate that exam performance is a strong predictor of success in real-world generation tasks, identifying a significant performance gap between model sizes and highlighting grammar as a primary linguistic hurdle.
The Evaluation Gap in Low-Resource Languages
Most LLMs are trained on a massive diet of English, leaving low-resource languages like Luxembourgish (spoken by approximately 600,000 people) as an afterthought. This neglect leads to "cross-lingual transfer" errors, where models erroneously apply German or French grammar rules to Luxembourgish.
Furthermore, researchers face an "evaluation gap": a lack of standardized benchmarks to measure progress. This is particularly critical for deploying models in privacy-sensitive local applications, as required by the Luxembourgish Data Protection Law. This paper investigates whether existing human language exams can serve as effective proxies for LLM performance.
Methodology
The researchers employed a systematic, zero-shot approach to evaluate 53 LLMs categorized by size: small (≤15B), medium (15B–200B), and large (>200B). The evaluation pipeline is illustrated in Figure 1.
- Exam Conversion: Official INLL exams (levels A1 to C2) were converted into Multiple Choice Questions (MCQs) for standardized, automated evaluation, resulting in a dataset of 629 questions.
- Linguistic Categorization: Questions were tagged as Vocabulary, Grammar, Reading Comprehension, or Conversation Comprehension.
- Downstream Validation: Models were tested on the LuxGen Benchmark, generating news headlines and Wikipedia-style descriptions to verify if exam scores correlate with practical utility.
- LLM-as-a-Judge: Claude 3.5 Sonnet was utilized to grade the quality of generated text, supplementing traditional metrics like METEOR and BERTScore.
Key Insights and Results
The Size Threshold and Linguistic Hurdles
A massive performance gulf exists between model sizes. Most small models (<15B) struggle to exceed random guessing on advanced exams, suggesting that emergent multilingual capabilities for low-resource languages require significant parameter counts. Across all models, Grammar proved to be the most difficult category, while Vocabulary was the highest-scoring. Models often struggle with the structural "glue" of the language, such as the n-rule (Eifeler Regel), which does not exist in neighboring high-resource languages like German.
Exams as a Performance Proxy
A central finding of the study is the strong positive correlation (Pearson’s $r$ up to 0.837) between a model's exam score and its performance on practical Natural Language Generation (NLG) tasks. As shown in Figure 2, higher exam scores consistently predict better performance across various metrics, including adequacy and grammar.
Top Performers
- Frontier Models: Claude 3.5 Sonnet and DeepSeek-R1 (671B) were the top performers. Claude led in most categories, while DeepSeek-R1 excelled on the most difficult C2-level exams.
- Small Model Standout: Gemma 3 (12B) significantly outperformed its peers in the small category, even rivaling some medium-sized models.
Conclusion and Future Outlook
While the study highlights the potential for using national proficiency exams as benchmarks, it also notes limitations such as potential data contamination in training sets and the need to explore fine-tuning for smaller models. However, the methodology provides a clear blueprint for testing other low-resource languages. To support ongoing research, the authors have released an interactive leaderboard to track the performance of new LLMs within the Luxembourgish NLP community.
Summary updated