Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks

Tatyana Iazykova, Denis Kapelyushnik, Olga Bystrova, Andrey Kutuzov

Introduction

Large Language Models (LLMs) currently dominate the Russian SuperGLUE (RSG) leaderboard. However, the extent to which these models truly comprehend the Russian language remains a subject of debate. Research in English NLP has frequently identified "annotation artifacts"—unintentional statistical cues such as sentence length or specific vocabulary—that allow models to achieve high scores through pattern matching rather than semantic understanding. This study investigates whether the RSG benchmark suffers from similar vulnerabilities by evaluating the performance of simple, rule-based heuristics against state-of-the-art neural models.

Methodology

The researchers employed a low-complexity approach involving zero machine learning to establish a baseline for the RSG tasks.

  1. Trivial Baselines: Three standard baselines were calculated: random choice, majority class, and random balanced choice (weighted by the training set's class distribution).
  2. Rule-Based Heuristics: By manually inspecting training and validation data, the authors identified shallow statistical regularities. These were implemented as simple Python functions. For instance, a rule might predict a label based solely on the word count of the hypothesis.
  3. Hybrid Evaluation: The test sets were evaluated using these heuristics, with trivial baselines serving as a fallback when no rules were triggered.

Heuristic Analysis

The authors developed specific heuristics for various tasks, such as the Russian Commitment Bank (RCB). These rules often relied on surface-level features like string matching or sentence length.

Table 1: Examples of rule-based heuristics for the RCB task, showing their coverage and accuracy.

As shown in the table above, simple rules like checking if the hypothesis is a substring of the premise or measuring the hypothesis length can yield significant coverage. Notably, a rule targeting the "neutral" label based on a specific hypothesis length range (between 4 and 8 words) achieved a 70% correctness rate, demonstrating how easily a model could exploit such artifacts.

Statistical Artifacts: The Length Bias

A key finding of the study is the strong correlation between sentence length and target labels in several RSG tasks.

Figure 1: Distribution of word counts for hypotheses (left) and premises (right) across different labels.

The boxplots illustrate that the "Number of words" is not uniformly distributed across classes. In many instances, the median length of a premise or hypothesis varies significantly depending on the label, providing a clear non-semantic signal that models can leverage to "cheat" on the task.

Key Results and Insights

The heuristic-based approach achieved an average RSG score of 0.468, outperforming RuGPT3-Small and remaining competitive with larger models like RuGPT3-Medium and RuBERT.

Specific task highlights include:

  • RWSD (Winograd Schema): The majority class baseline matched the state-of-the-art performance (0.669 accuracy), suggesting that complex models were simply guessing the most frequent label rather than performing pronoun disambiguation.
  • DaNetQA: Heuristics based on question structure (e.g., "Does it belong to...") achieved 0.642 accuracy, far exceeding the random baseline of 0.494.
  • Data Leakage: The study found that class imbalances in the training set are often mirrored in the test set. A random weighted baseline scores 0.538, significantly higher than a purely random baseline (0.472), which artificially inflates model performance.

Conclusion and Recommendations

The success of these simple heuristics suggests that many top-performing models on the RSG leaderboard may be exploiting shallow statistical cues rather than demonstrating genuine linguistic reasoning. To improve the robustness of the benchmark, the authors recommend balancing test sets to eliminate data leakage, removing obvious length-based cues, and incorporating adversarial datasets specifically designed to trick models that rely on shallow heuristics.

Summary updated

Abstract

Leaderboards like SuperGLUE are seen as important incentives for active development of NLP, since they provide standard benchmarks for fair comparison of modern language models. They have driven the world's best engineering teams as well as their resources to collaborate and solve a set of tasks for general language understanding. Their performance scores are often claimed to be close to or even higher than the human performance. These results encouraged more thorough analysis of whether the benchmark datasets featured any statistical cues that machine learning based language models can exploit. For English datasets, it was shown that they often contain annotation artifacts. This allows solving certain tasks with very simple rules and achieving competitive rankings.
BibTeX
@inproceedings{iazykova2021unreasonable,
  author = {Tatyana Iazykova and Denis Kapelyushnik and Olga Bystrova and Andrey Kutuzov},
  title = {{Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks}},
  booktitle = {Kompʹûternaâ},
  year = {2021},
  doi = {10.28995/2075-7182-2021-20-302-317},
}
Figure