TL;DR
The authors introduce BLiMP-RO, a comprehensive, human-validated benchmark for evaluating the grammatical knowledge of Romanian language models using linguistic minimal pairs. Their experiments reveal that while model performance scales predictably with data size, the composition of training data (e.g., books vs. video captions) and model architecture (GPT-BERT vs. GPT-2) significantly influence how effectively a model acquires complex syntax.
What Problem Does This Paper Address?
Evaluating whether a Large Language Model (LLM) actually "understands" grammar is tricky. Standard benchmarks often conflate general predictive power with specific syntactic knowledge. To isolate the latter, researchers use minimal pairs: two sentences that differ by only one grammatical feature (e.g., "The cat eats" vs. "The cat eat"). If a model assigns a higher probability to the correct version, it demonstrates grammatical awareness.
For Romanian, existing tools were sparse. The previous standard, MultiBLiMP, was derived automatically from treebanks, lacked human validation, and ignored unique Romanian features like differential object marking (DOM) and clitic doubling. BLiMP-RO fills this gap with a linguist-curated, human-verified dataset designed specifically for the nuances of the Romanian language.
Methods
The researchers employed a two-pronged approach:
- Benchmark Construction: They developed 50 grammatical phenomena across ten domains (see the breakdown in the table below). They created an initial hand-crafted set of 500 items for human validation and an extended set of nearly 6,000 items generated semi-automatically using Claude Opus 5.5 and then human-validated.
- Developmental Training: Following the "BabyLM" philosophy, they trained models from scratch on small, "developmentally motivated" datasets (1M to 10M words) rather than massive web crawls. They compared two architectures: GPT-2 (causal/left-to-right) and GPT-BERT (a hybrid using both causal and masked objectives).
- Scoring: They used the Syntactic Log-Odds Ratio (SLOR) to score sentences. SLOR is effective because it adjusts for sentence length and word frequency, preventing the model from simply preferring shorter or more common sentences.
Datasets
- BLiMP-RO (New): 6,435 total minimal pairs (493 hand-crafted, 5,942 semi-automatic).
- Romanian BabyBabelLM: A mix of child-directed speech, children's books, and subtitles.
- YouTube Kids Captions (New): A curated 10-million-word corpus of Romanian captions from child-targeted videos.
- MultiBLiMP: Used as a secondary baseline for comparison.
Key Insights and Ideas
- The Scaling Law of Grammar: Grammatical accuracy improves linearly with the logarithm of the training set size. Specifically, models gained about 9 percentage points in accuracy for every tenfold increase in data.
- Architecture Matters at Small Scales: GPT-BERT consistently outperformed GPT-2 across almost all data sizes and phenomena. This suggests that bidirectional context or masked objectives might be more efficient for learning syntax when data is scarce.
- Quality Over Quantity: At matched sizes, models trained on the BabyBabelLM (which includes books) outperformed those trained on raw video captions. The authors suggest that the richer vocabulary and more formal structures found in books provide a better "training signal" for grammar than the more repetitive language of video captions.
- Hard vs. Easy Domains: Even pretrained giants struggle with certain areas. While models are excellent at simple subject-verb agreement, they find Binding Theory (how pronouns refer back to nouns) and Syntactic Extraction (complex "who/what" questions) much more challenging.
Key Results
- Human Baseline: Native speakers achieved 96% accuracy on the benchmark, setting a high bar for models.
- Pretrained Models: Romanian-specific models like RoBERT (92%) and RoGPT2 (90%) performed well, significantly outperforming larger multilingual models like EuroLLM (82%).
- Small Model Performance: Models trained on just 10M words of captions reached roughly 73-77% accuracy. While far from human levels, this shows significant grammatical emergence from very limited input.
- Scaling Consistency: The scaling behavior was remarkably consistent across both the new BLiMP-RO and the older MultiBLiMP benchmarks, as seen in the figure below.
Limitations and Future Work
The authors acknowledge that forced-choice tasks are a proxy; they measure relative preference rather than absolute sentence acceptability. Furthermore, while they identified that certain data types are better, the exact mechanisms—why a book is better than a transcript for a neural network—remain a "black box."
Future work could involve investigating which specific properties of training input (like syntactic complexity or lexical diversity) drive the acquisition of specific grammatical rules. There is also room to expand the benchmark to include more "preference" items where both sentences are grammatical but one is more natural, further challenging the nuances of model "intuition."
Summary updated