Introduction
Large Language Models (LLMs) have revolutionized natural language processing, yet their mastery of low-resource languages remains questionable. While a model might produce a fluent translation in Luxembourgish, does it truly understand the underlying syntactic and morphological rules? This paper investigates the discrepancy between translation fluency and grammatical competence using a novel, grammar-book-guided probing pipeline.
Methodology: The GBGP Pipeline
We propose the Grammar-Book–Guided Probing (GBGP) pipeline to bridge the gap between descriptive linguistics and automated evaluation. This framework systematically transforms physical grammar rules into a structured testing environment.
The pipeline operates in four distinct stages:
- Material Inspector: Utilizing OCR and LLM-based cleaning, we extract rules, tables, and examples from authoritative Luxembourgish grammar books.
- Phrasing Atelier: This stage distills extracted data into specific "grammar points" and generates English-Luxembourgish sentence pairs designed to trigger those specific rules.
- Twin Forge: A critical "back-checking" phase where the model generates Minimal Pairs—two sentences identical except for one grammatical feature (one correct, one incorrect). This tests the model's ability to discern subtle rule violations.
- Proof Stand: The final evaluation suite consisting of four tasks: matching sentences to grammar descriptions (Tasks 1-2), inferring multiple rules from paragraphs (Task 3), and identifying the correct sentence in a minimal pair (Task 4).
The MIXED Benchmark
Our evaluation utilizes the new MIXED benchmark, totaling 3,352 pairs. This dataset includes G2040 (sentences directly from grammar books), FLORES200 (standard evaluation sentences), and LONG300 (complex, long-form web sentences). This variety ensures that models are tested on both idealized rules and real-world usage.
Results and Key Insights
Our analysis reveals that translation performance is a poor proxy for grammatical understanding.
The Fluency-Grammar Gap
We observed a weak positive correlation between standard translation metrics (BLEU, CometScore) and performance on our grammar tasks. As shown in the correlation heatmaps below, a model's ability to convey meaning does not guarantee adherence to morphological rules.
Scaling and Reasoning
While scaling model size (e.g., from Llama 3-1B to 70B) drastically improves translation scores, grammatical accuracy on Task 4 (Minimal Pairs) remains stubbornly low. Even top-tier models like GPT-5 only achieved 61% accuracy on minimal pairs, barely outperforming random chance. Conversely, "Thinking" models like Qwen3-Next-80B-Thinking showed superior performance, suggesting that grammatical adherence is a logic-intensive task better served by chain-of-thought reasoning than raw parameter count.
Phonological Awareness
LLMs struggle significantly with phonology-based rules, such as the Eifeler Regel (dropping the terminal "n" based on the following word's sound). Because models operate on tokens rather than acoustic representations, they lack the "phonological awareness" required for these specific linguistic nuances.
Conclusion
This research demonstrates that LLMs often act as "stochastic parrots" in low-resource settings, achieving surface-level fluency without a deep grasp of grammar. The GBGP pipeline provides a scalable method for identifying these gaps, highlighting the need for models that integrate logical reasoning and phonological awareness to achieve true linguistic competence.
Summary updated