Abstract
In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open-source framework for evaluating models, and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of comparing multilingual models in the adapted diagnostic test set and offer the first steps to further expanding or assessing state-of-the-art models independently of language.
Introduction
For years, the "gold standard" for evaluating how well an AI understands language has been dominated by English-centric benchmarks like GLUE and SuperGLUE. However, language is not a monolith. Russian, with its rich inflectional morphology, flexible word order, and unique syntactic structures, presents challenges that English-focused models might overlook.
Before this research, there was no unified, difficult "yardstick" to measure the progress of Russian Natural Language Understanding (NLU). Researchers often relied on translated datasets or simple classification tasks that didn't capture the nuance of the language. RussianSuperGLUE fills this gap by providing a standardized, multi-task framework to push the boundaries of Russian language modeling.
Methodology and Datasets
The authors adopted the SuperGLUE paradigm, which favors tasks that require complex reasoning rather than simple pattern matching. The benchmark includes nine tasks categorized into six groups:
- Diagnostics: LiDiRus (Linguistic Diagnostic for Russian), a hand-crafted test to probe specific linguistic phenomena like negation and quantifiers.
- Common Sense: RUSSE (Word sense disambiguation) and PARus (Causal reasoning).
- Natural Language Inference (NLI): TERRa (Textual entailment) and RCB (Russian Commitment Bank).
- Reasoning: RWSD (Winograd Schema Challenge for coreference resolution).
- Machine Reading: MuSeRC (Multi-sentence reading comprehension) and RuCoS (Cloze-style reasoning on news articles).
- World Knowledge: DaNetQA (Yes/no questions based on Wikipedia).
While some datasets were translated and professionally edited (like PARus and RWSD), others were built from scratch using Russian-specific sources like the Taiga corpus and Russian National Corpus. To establish a performance ceiling, the authors used crowd-workers via the Toloka platform to provide a human baseline.
Key Results and Analysis
The evaluation of baseline models—including a "naive" TF-IDF model, multilingual BERT (MultiBERT), and a specialized Russian BERT (RuBERT)—revealed several critical insights.
The Human-Model Gap
A primary finding is the significant gap between the best-performing model (RuBERT at 0.546) and human performance (0.802). This 25-point difference suggests that while models are improving at processing Russian text, they still lack true reasoning capabilities.
Monolingual vs. Multilingual Performance
RuBERT generally outperformed MultiBERT, particularly in NLI and reading comprehension tasks. This highlights the value of pre-training on large-scale, language-specific corpora to capture unique morphological nuances. However, an interesting anomaly occurred in the RUSSE task (word sense disambiguation), where models actually outperformed humans. This may be because the task format is susceptible to statistical patterns that transformers excel at, whereas humans may find the specific context-matching ambiguous.
Linguistic Bias in Architecture
Diagnostic tests showed that BERT-style models perform better on English-centric syntactic categories than Russian ones. For example, performance on categories like Datives or PA Structure differs significantly between languages. This hints that current transformer architectures might be implicitly "optimized" for English syntax.
Conclusion and Future Work
RussianSuperGLUE establishes a rigorous new standard for Russian NLU. The authors acknowledge that as models continue to evolve, these tasks may eventually be "solved." Consequently, future work will focus on increasing complexity by adding long-form document understanding and generative (seq2seq) challenges. Additionally, there are plans to incorporate external structured knowledge via knowledge graphs and to introduce industrial metrics like memory usage and inference speed to the leaderboard.
Summary updated