Abstract
Linguistic Acceptability (LA) serves as a critical measure of a language model's ability to distinguish between well-formed and ill-formed sentences. While benchmarks like CoLA have advanced this research for English, morphologically rich languages with free word order, such as Russian, have lacked similar large-scale resources. This paper introduces RuCoLA (Russian Corpus of Linguistic Acceptability), a benchmark of 13,445 sentences designed to evaluate the grammatical competence of state-of-the-art models. Our findings reveal a significant performance gap between humans and models, particularly in morphology and semantics, highlighting the limitations of current architectures in mastering the nuances of the Russian language.
Introduction
In Natural Language Processing (NLP), the ability of a model to judge the "acceptability" of a sentence—whether it sounds natural and correct to a native speaker—is a proxy for its underlying linguistic understanding. For years, the English CoLA benchmark has been the standard for this task. However, Russian presents unique challenges that English-centric benchmarks cannot capture: a complex system of cases and word endings (morphology) and a relatively fluid sentence structure (syntax).
RuCoLA addresses this gap by providing a comprehensive dataset that tests both theoretical linguistic phenomena and the practical errors produced by modern generative AI. By comparing human judgments against various Transformer-based models, we can pinpoint exactly where current AI falls short of native-level fluency.
Methodology
Dataset Composition
The researchers constructed RuCoLA using a two-pronged strategy to ensure both breadth and depth:
- In-domain Data (9.8k sentences): These examples were manually curated from linguistic textbooks and academic literature. They focus on specific, often subtle, linguistic phenomena such as complex syntax, the semantics of tense, and the nuances of negation.
- Out-of-domain Data (3.6k sentences): To reflect real-world AI limitations, this set includes sentences generated by nine different machine translation and paraphrase models. This allows the benchmark to measure how well a classifier can detect "hallucinations" or unnatural phrasing produced by other models.
Annotation Pipeline
To ensure high data quality, the authors employed a rigorous two-stage process on the Toloka crowdsourcing platform:
- Stage 1 (Binary Labeling): Native speakers provided initial labels, marking sentences as either acceptable or unacceptable.
- Stage 2 (Expert Categorization): Linguistics students reviewed the unacceptable sentences, categorizing the errors into four distinct domains: Morphology, Syntax, Semantics, and Hallucinations.
Of the 13,445 total sentences, approximately 71.8% are acceptable, while 28.2% contain labeled errors.
Experimental Results
The study benchmarked several prominent models, including monolingual Russian models (ruBERT, ruRoBERTa, ruT5) and cross-lingual models (XLM-R, RemBERT). Performance was measured using the Matthews Correlation Coefficient (MCC), a robust metric for imbalanced classification tasks.
The Human-Model Gap
The results show a clear hierarchy in performance:
- Human Baseline: 0.63 MCC
- Best Model (ruRoBERTa-large): 0.54 MCC
While the 0.09 gap might seem small, a deeper look at error recall reveals a stark disparity. For morphological errors—such as incorrect word derivations—humans achieved 81.5% recall, whereas ruRoBERTa-large managed only 44.4%. Similarly, in the realm of semantics, humans outperformed the best model by 20 percentage points (57.1% vs. 37.1%).
Key Insights
- The Morphology Gap: Russian models struggle significantly with the internal logic of word-building. Despite their large parameter counts, they often fail to recognize non-existent words or incorrect case endings that are obvious to native speakers.
- Syntactic Proficiency: Models performed best on syntactic violations, such as word order and agreement. This suggests that the attention mechanisms in Transformer architectures are well-suited for capturing structural patterns.
- The Failure of Zero-Shot Transfer: Attempts to train models on English (CoLA) or Italian (ItaCoLA) data and apply them to Russian yielded poor results. This confirms that linguistic acceptability is deeply language-specific and cannot be easily transferred across different linguistic families.
Discussion and Practical Utility
Beyond its value as a research benchmark, RuCoLA has immediate practical applications. The out-of-domain dataset demonstrates that acceptability classifiers can serve as effective "filters" for generative models. By integrating a RuCoLA-trained classifier into an AI pipeline, developers can catch and remove nonsensical or ungrammatical text before it reaches the end-user.
However, the study also notes limitations. The reliance on linguistic literature for in-domain data may introduce a bias toward "textbook" examples that do not always reflect the informal nature of everyday speech. Furthermore, the debate continues regarding whether expert linguists or "naive" native speakers provide the most representative judgments of acceptability.
Conclusion
RuCoLA represents a major step forward for Russian NLP, providing a rigorous framework for evaluating the grammatical integrity of language models. While current models show promise in handling syntax, the "morphology gap" remains a significant hurdle. Future work will focus on using RuCoLA as a training signal to help generative models produce more natural, linguistically accurate Russian text, eventually closing the gap between machine output and human fluency.
Summary updated