RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark

Tatiana Shavrina, Alena Fenogenova, Anton A. Emelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, Andrey Evlampiev

Abstract

In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open-source framework for evaluating models, and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of comparing multilingual models in the adapted diagnostic test set and offer the first steps to further expanding or assessing state-of-the-art models independently of language.

Introduction

For years, the "gold standard" for evaluating how well an AI understands language has been dominated by English-centric benchmarks like GLUE and SuperGLUE. However, language is not a monolith. Russian, with its rich inflectional morphology, flexible word order, and unique syntactic structures, presents challenges that English-focused models might overlook.

Before this research, there was no unified, difficult "yardstick" to measure the progress of Russian Natural Language Understanding (NLU). Researchers often relied on translated datasets or simple classification tasks that didn't capture the nuance of the language. RussianSuperGLUE fills this gap by providing a standardized, multi-task framework to push the boundaries of Russian language modeling.

Methodology and Datasets

The authors adopted the SuperGLUE paradigm, which favors tasks that require complex reasoning rather than simple pattern matching. The benchmark includes nine tasks categorized into six groups:

  1. Diagnostics: LiDiRus (Linguistic Diagnostic for Russian), a hand-crafted test to probe specific linguistic phenomena like negation and quantifiers.
  2. Common Sense: RUSSE (Word sense disambiguation) and PARus (Causal reasoning).
  3. Natural Language Inference (NLI): TERRa (Textual entailment) and RCB (Russian Commitment Bank).
  4. Reasoning: RWSD (Winograd Schema Challenge for coreference resolution).
  5. Machine Reading: MuSeRC (Multi-sentence reading comprehension) and RuCoS (Cloze-style reasoning on news articles).
  6. World Knowledge: DaNetQA (Yes/no questions based on Wikipedia).

While some datasets were translated and professionally edited (like PARus and RWSD), others were built from scratch using Russian-specific sources like the Taiga corpus and Russian National Corpus. To establish a performance ceiling, the authors used crowd-workers via the Toloka platform to provide a human baseline.

Key Results and Analysis

The evaluation of baseline models—including a "naive" TF-IDF model, multilingual BERT (MultiBERT), and a specialized Russian BERT (RuBERT)—revealed several critical insights.

The Human-Model Gap

A primary finding is the significant gap between the best-performing model (RuBERT at 0.546) and human performance (0.802). This 25-point difference suggests that while models are improving at processing Russian text, they still lack true reasoning capabilities.

Table 1: Performance comparison across datasets for baseline models and human evaluation.

Monolingual vs. Multilingual Performance

RuBERT generally outperformed MultiBERT, particularly in NLI and reading comprehension tasks. This highlights the value of pre-training on large-scale, language-specific corpora to capture unique morphological nuances. However, an interesting anomaly occurred in the RUSSE task (word sense disambiguation), where models actually outperformed humans. This may be because the task format is susceptible to statistical patterns that transformers excel at, whereas humans may find the specific context-matching ambiguous.

Linguistic Bias in Architecture

Diagnostic tests showed that BERT-style models perform better on English-centric syntactic categories than Russian ones. For example, performance on categories like Datives or PA Structure differs significantly between languages. This hints that current transformer architectures might be implicitly "optimized" for English syntax.

Figure 1: Diagnostic heatmap comparing model performance on linguistic phenomena in English vs. Russian.

Conclusion and Future Work

RussianSuperGLUE establishes a rigorous new standard for Russian NLU. The authors acknowledge that as models continue to evolve, these tasks may eventually be "solved." Consequently, future work will focus on increasing complexity by adding long-form document understanding and generative (seq2seq) challenges. Additionally, there are plans to incorporate external structured knowledge via knowledge graphs and to introduce industrial metrics like memory usage and inference speed to the leaderboard.

Summary updated

Abstract

In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open-source framework for evaluating models (https://github.com/RussianNLP/RussianSuperGLUE), and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of comparing multilingual models in the adapted diagnostic test set and offer the first steps to further expanding or assessing state-of-the-art models independently of language.
BibTeX
@inproceedings{shavrina2020russiansuperglue,
  author = {Tatiana Shavrina and Alena Fenogenova and Anton A. Emelyanov and Denis Shevelev and Ekaterina Artemova and Valentin Malykh and Vladislav Mikhailov and Maria Tikhonova and Andrey Chertok and Andrey Evlampiev},
  title = {{RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.381},
}
Figure