RuCoLA: Russian Corpus of Linguistic Acceptability

Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin, Alena Pestova, Ivan Smurov, Ekaterina Artemova

Introduction

In the field of linguistics, "acceptability" refers to whether a native speaker perceives a sentence as natural and well-formed. While the Corpus of Linguistic Acceptability (CoLA) has long served as the benchmark for English, the Russian language—characterized by its rich morphology and flexible word order—has lacked a comparable high-quality resource.

RuCoLA (Russian Corpus of Linguistic Acceptability) addresses this gap. It is a large-scale benchmark comprising 13,445 sentences designed to evaluate how effectively Natural Language Processing (NLP) models can judge the linguistic integrity of Russian text. Unlike previous datasets that focused solely on theoretical examples, RuCoLA bridges the gap between formal grammar and the practical, often nonsensical errors produced by modern AI models.

Methodology: A Two-Pronged Approach

The researchers constructed RuCoLA using two distinct data sources to capture a wide spectrum of linguistic phenomena:

  1. In-domain Data (9.8k sentences): These were manually curated from fundamental Russian linguistic textbooks and academic papers. This subset includes "minimal pairs"—sets of sentences where one is grammatically correct and the other contains a specific, isolated violation.
  2. Out-of-domain Data (3.6k sentences): To capture the "messy" errors typical of modern AI, the researchers used nine different machine translation and paraphrase models (such as ruGPT-2 and ruT5) to generate text.

Annotation followed a rigorous two-stage process. First, native speakers on a crowdsourcing platform performed binary classification (Acceptable vs. Unacceptable). Second, expert philology students categorized the unacceptable sentences into four specific types: Morphology, Syntax, Semantics, and Hallucinations.

Dataset Composition and Error Analysis

The distribution of error types varies significantly between the in-domain and out-of-domain subsets, reflecting the different nature of theoretical linguistic errors versus machine-generated artifacts.

Distribution of linguistic error types across the RuCoLA dataset and its sub-domains.

As shown in the figure above, syntax errors constitute the largest portion of the dataset overall. Notably, "Hallucinations"—nonsensical repetitions or logical breaks—are exclusive to the out-of-domain data, highlighting the unique challenges posed by generative AI outputs compared to traditional linguistic examples.

Key Insights and Model Performance

The benchmarking of various models, from simple baselines to state-of-the-art Transformers like ruRoBERTa and RemBERT, revealed several critical findings:

  • The Morphology Barrier: While models are relatively adept at identifying syntax (word order) errors, they struggle significantly with morphology. In Russian, where a single character change in a word ending can alter the entire grammatical role of a word, models still lack the fine-grained sensitivity of a native speaker.
  • The Human-Machine Gap: There remains a substantial performance gap. The top-performing model, ruRoBERTa-large, achieved a Matthews Correlation Coefficient (MCC) of 0.54, whereas human native speakers scored 0.63.
  • Detection Strengths: Models were most successful at identifying Hallucinations (achieving up to 77% recall) but failed most frequently on Morphology (with recall dropping as low as 17% for some architectures).
  • Cross-Lingual Limitations: Experiments with "zero-shot" transfer—training on English CoLA and testing on RuCoLA—yielded poor results for in-domain data. This suggests that linguistic acceptability is deeply tied to the specific structural quirks of a language rather than being a universal trait that models can easily transfer.

Conclusion

RuCoLA establishes a challenging new standard for Russian NLP. The results indicate that while modern Transformers are evolving, they still lack a deep "feel" for the complex morphological and semantic nuances of Russian. Future work may involve using RuCoLA-trained classifiers as filters to improve the output of generative models, effectively providing AI with a sophisticated "grammar checker" that understands the specific complexities of the Russian language.

Summary updated

BibTeX
@inproceedings{mikhailov2022rucola,
  author = {Vladislav Mikhailov and Tatiana Shamardina and Max Ryabinin and Alena Pestova and Ivan Smurov and Ekaterina Artemova},
  title = {{RuCoLA: Russian Corpus of Linguistic Acceptability}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=IgKU_3MCdnM},
}
Figure