BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian

Radu-Stefan Popa, Valentin Feroiu, Adina Camelia Bleotu, Sergiu NisioiarXiv 2026 · arXiv

TL;DR

The authors introduce BLiMP-RO, a comprehensive, human-validated benchmark for evaluating the grammatical knowledge of Romanian language models using linguistic minimal pairs. Their experiments reveal that while model performance scales predictably with data size, the composition of training data (e.g., books vs. video captions) and model architecture (GPT-BERT vs. GPT-2) significantly influence how effectively a model acquires complex syntax.

What Problem Does This Paper Address?

Evaluating whether a Large Language Model (LLM) actually "understands" grammar is tricky. Standard benchmarks often conflate general predictive power with specific syntactic knowledge. To isolate the latter, researchers use minimal pairs: two sentences that differ by only one grammatical feature (e.g., "The cat eats" vs. "The cat eat"). If a model assigns a higher probability to the correct version, it demonstrates grammatical awareness.

For Romanian, existing tools were sparse. The previous standard, MultiBLiMP, was derived automatically from treebanks, lacked human validation, and ignored unique Romanian features like differential object marking (DOM) and clitic doubling. BLiMP-RO fills this gap with a linguist-curated, human-verified dataset designed specifically for the nuances of the Romanian language.

Methods

The researchers employed a two-pronged approach:

  1. Benchmark Construction: They developed 50 grammatical phenomena across ten domains (see the breakdown in the table below). They created an initial hand-crafted set of 500 items for human validation and an extended set of nearly 6,000 items generated semi-automatically using Claude Opus 5.5 and then human-validated.
  2. Developmental Training: Following the "BabyLM" philosophy, they trained models from scratch on small, "developmentally motivated" datasets (1M to 10M words) rather than massive web crawls. They compared two architectures: GPT-2 (causal/left-to-right) and GPT-BERT (a hybrid using both causal and masked objectives).
  3. Scoring: They used the Syntactic Log-Odds Ratio (SLOR) to score sentences. SLOR is effective because it adjusts for sentence length and word frequency, preventing the model from simply preferring shorter or more common sentences.
The 50 phenomena covered by BLiMP-RO across 10 domains.

Datasets

  • BLiMP-RO (New): 6,435 total minimal pairs (493 hand-crafted, 5,942 semi-automatic).
  • Romanian BabyBabelLM: A mix of child-directed speech, children's books, and subtitles.
  • YouTube Kids Captions (New): A curated 10-million-word corpus of Romanian captions from child-targeted videos.
  • MultiBLiMP: Used as a secondary baseline for comparison.

Key Insights and Ideas

  • The Scaling Law of Grammar: Grammatical accuracy improves linearly with the logarithm of the training set size. Specifically, models gained about 9 percentage points in accuracy for every tenfold increase in data.
  • Architecture Matters at Small Scales: GPT-BERT consistently outperformed GPT-2 across almost all data sizes and phenomena. This suggests that bidirectional context or masked objectives might be more efficient for learning syntax when data is scarce.
  • Quality Over Quantity: At matched sizes, models trained on the BabyBabelLM (which includes books) outperformed those trained on raw video captions. The authors suggest that the richer vocabulary and more formal structures found in books provide a better "training signal" for grammar than the more repetitive language of video captions.
  • Hard vs. Easy Domains: Even pretrained giants struggle with certain areas. While models are excellent at simple subject-verb agreement, they find Binding Theory (how pronouns refer back to nouns) and Syntactic Extraction (complex "who/what" questions) much more challenging.
Figure 1: Accuracy (%) on each phenomenon, grouped by the domains of Table 3. Circles are people, squares the mean of the two pretrained Romanian models and triangles the mean of GPT-2 and GPT-BERT trained on captions 1M, BBL 1M and captions 10M (Table 2). People are on the 508 items, models on all pairs, with the 5,942 pairs that humans accepted. A gray line joins the three marks of a phenomenon. Within a domain, rows go from the lowest human accuracy at the top to the highest at the bottom. Names are shortened from Table 1; anaphor agreement and the C-command condition test emphatic reflexives. The axis starts at 30. Table 7 gives the values of each model.

Key Results

  • Human Baseline: Native speakers achieved 96% accuracy on the benchmark, setting a high bar for models.
  • Pretrained Models: Romanian-specific models like RoBERT (92%) and RoGPT2 (90%) performed well, significantly outperforming larger multilingual models like EuroLLM (82%).
  • Small Model Performance: Models trained on just 10M words of captions reached roughly 73-77% accuracy. While far from human levels, this shows significant grammatical emergence from very limited input.
  • Scaling Consistency: The scaling behavior was remarkably consistent across both the new BLiMP-RO and the older MultiBLiMP benchmarks, as seen in the figure below.
Figure 2: Accuracy under SLOR of caption models against training words, on (a) BLiMP-RO, all 6,435 pairs: the 493 items and the 5,942 pairs that humans accepted, and (b) MultiBLiMP. The 15 nested caption sets have 0.1, 0.2, 0.3, 0.4, 0.6, 0.8, 1.2, 1.6, 2, 2.5, 3.2, 4, 5, 6.4 and 8.2 million training words, without vali- dation text; the last is the whole caption pool (captions 10M in Table 2). All models use the same validation set of 160,000 words, half captions and half BabyBa- belLM, only to choose the checkpoint. Small points are runs, large points their means, lines ordinary least- squares fits of the means, and labels their gain in points per tenfold increase.

Limitations and Future Work

The authors acknowledge that forced-choice tasks are a proxy; they measure relative preference rather than absolute sentence acceptability. Furthermore, while they identified that certain data types are better, the exact mechanisms—why a book is better than a transcript for a neural network—remain a "black box."

Future work could involve investigating which specific properties of training input (like syntactic complexity or lexical diversity) drive the acquisition of specific grammatical rules. There is also room to expand the benchmark to include more "preference" items where both sentences are grammatical but one is more natural, further challenging the nuances of model "intuition."

Summary updated

Abstract

Evaluating grammatical knowledge in language models requires benchmarks that cover diverse linguistic phenomena and are validated against human judgments. We introduce BLiMP-RO, a Romanian minimal-pair benchmark covering 50 grammatical phenomena across ten domains. It comprises 493 linguist-constructed sentence pairs, evaluated by native speakers in five experiments, and 5,942 additional semi-automatically generated, human-validated pairs. We further investigate grammatical generalization under limited linguistic input by training GPT-2 and GPT-BERT models from scratch on Romanian BabyBabelLM data and a newly collected corpus of 10 million words from child-targeted video captions. Increasing caption training data yields approximately 9 percentage points of accuracy improvement per tenfold increase in corpus size. At matched data sizes, BabyBabelLM generally outperforms video captions, while GPT-BERT consistently outperforms GPT-2. Our findings highlight how training data quantity, composition, and model architecture shape grammatical generalization in Romanian.
BibTeX
@misc{popa2026blimp,
  author = {Radu-Stefan Popa and Valentin Feroiu and Adina Camelia Bleotu and Sergiu Nisioi},
  title = {{BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian}},
  year = {2026},
  eprint = {2611.17836},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17836},
}
Figure