Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish

Lujun Li, Yewei Song, Lama Sleem, Yiqun Wang, Yangjie Xu, Cedric Lothritz, Niccolò Gentile, Radu State, Tegawendé F. Bissyandé, Jacques KleinarXiv 2025 · DOI · arXiv

Introduction

Large Language Models (LLMs) have revolutionized natural language processing, yet their mastery of low-resource languages remains questionable. While a model might produce a fluent translation in Luxembourgish, does it truly understand the underlying syntactic and morphological rules? This paper investigates the discrepancy between translation fluency and grammatical competence using a novel, grammar-book-guided probing pipeline.

Methodology: The GBGP Pipeline

We propose the Grammar-Book–Guided Probing (GBGP) pipeline to bridge the gap between descriptive linguistics and automated evaluation. This framework systematically transforms physical grammar rules into a structured testing environment.

The four-stage Grammar-Book-Guided Probing (GBGP) pipeline, moving from raw material to final evaluation.

The pipeline operates in four distinct stages:

  1. Material Inspector: Utilizing OCR and LLM-based cleaning, we extract rules, tables, and examples from authoritative Luxembourgish grammar books.
  2. Phrasing Atelier: This stage distills extracted data into specific "grammar points" and generates English-Luxembourgish sentence pairs designed to trigger those specific rules.
  3. Twin Forge: A critical "back-checking" phase where the model generates Minimal Pairs—two sentences identical except for one grammatical feature (one correct, one incorrect). This tests the model's ability to discern subtle rule violations.
  4. Proof Stand: The final evaluation suite consisting of four tasks: matching sentences to grammar descriptions (Tasks 1-2), inferring multiple rules from paragraphs (Task 3), and identifying the correct sentence in a minimal pair (Task 4).

The MIXED Benchmark

Our evaluation utilizes the new MIXED benchmark, totaling 3,352 pairs. This dataset includes G2040 (sentences directly from grammar books), FLORES200 (standard evaluation sentences), and LONG300 (complex, long-form web sentences). This variety ensures that models are tested on both idealized rules and real-world usage.

Results and Key Insights

Our analysis reveals that translation performance is a poor proxy for grammatical understanding.

The Fluency-Grammar Gap

We observed a weak positive correlation between standard translation metrics (BLEU, CometScore) and performance on our grammar tasks. As shown in the correlation heatmaps below, a model's ability to convey meaning does not guarantee adherence to morphological rules.

Correlation heatmaps (Pearson and Spearman) illustrating the weak relationship between translation metrics and grammar task performance.

Scaling and Reasoning

While scaling model size (e.g., from Llama 3-1B to 70B) drastically improves translation scores, grammatical accuracy on Task 4 (Minimal Pairs) remains stubbornly low. Even top-tier models like GPT-5 only achieved 61% accuracy on minimal pairs, barely outperforming random chance. Conversely, "Thinking" models like Qwen3-Next-80B-Thinking showed superior performance, suggesting that grammatical adherence is a logic-intensive task better served by chain-of-thought reasoning than raw parameter count.

Phonological Awareness

LLMs struggle significantly with phonology-based rules, such as the Eifeler Regel (dropping the terminal "n" based on the following word's sound). Because models operate on tokens rather than acoustic representations, they lack the "phonological awareness" required for these specific linguistic nuances.

Conclusion

This research demonstrates that LLMs often act as "stochastic parrots" in low-resource settings, achieving surface-level fluency without a deep grasp of grammar. The GBGP pipeline provides a scalable method for identifying these gaps, highlighting the need for models that integrate logical reasoning and phonological awareness to achieve true linguistic competence.

Summary updated

Abstract

Grammar refers to the system of rules that governs the structural organization and the semantic relations among linguistic units such as sentences, phrases, and words within a given language. In natural language processing, there remains a notable scarcity of grammar focused evaluation protocols, a gap that is even more pronounced for low-resource languages. Moreover, the extent to which large language models genuinely comprehend grammatical structure, especially the mapping between syntactic structures and meanings, remains under debate. To investigate this issue, we propose a Grammar Book Guided evaluation pipeline intended to provide a systematic and generalizable framework for grammar evaluation consisting of four key stages, and in this work we take Luxembourgish as a case study. The results show a weak positive correlation between translation performance and grammatical understanding, indicating that strong translations do not necessarily imply deep grammatical competence. Larger models perform well overall due to their semantic strength but remain weak in morphology and syntax, struggling particularly with Minimal Pair tasks, while strong reasoning ability offers a promising way to enhance their grammatical understanding.
BibTeX
@misc{li2025do,
  author = {Lujun Li and Yewei Song and Lama Sleem and Yiqun Wang and Yangjie Xu and Cedric Lothritz and Niccolò Gentile and Radu State and Tegawendé F. Bissyandé and Jacques Klein},
  title = {{Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish}},
  year = {2025},
  eprint = {2510.24856},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.24856},
}
Figure