Introduction
Large Language Models (LLMs) currently dominate the Russian SuperGLUE (RSG) leaderboard. However, the extent to which these models truly comprehend the Russian language remains a subject of debate. Research in English NLP has frequently identified "annotation artifacts"—unintentional statistical cues such as sentence length or specific vocabulary—that allow models to achieve high scores through pattern matching rather than semantic understanding. This study investigates whether the RSG benchmark suffers from similar vulnerabilities by evaluating the performance of simple, rule-based heuristics against state-of-the-art neural models.
Methodology
The researchers employed a low-complexity approach involving zero machine learning to establish a baseline for the RSG tasks.
- Trivial Baselines: Three standard baselines were calculated: random choice, majority class, and random balanced choice (weighted by the training set's class distribution).
- Rule-Based Heuristics: By manually inspecting training and validation data, the authors identified shallow statistical regularities. These were implemented as simple Python functions. For instance, a rule might predict a label based solely on the word count of the hypothesis.
- Hybrid Evaluation: The test sets were evaluated using these heuristics, with trivial baselines serving as a fallback when no rules were triggered.
Heuristic Analysis
The authors developed specific heuristics for various tasks, such as the Russian Commitment Bank (RCB). These rules often relied on surface-level features like string matching or sentence length.
As shown in the table above, simple rules like checking if the hypothesis is a substring of the premise or measuring the hypothesis length can yield significant coverage. Notably, a rule targeting the "neutral" label based on a specific hypothesis length range (between 4 and 8 words) achieved a 70% correctness rate, demonstrating how easily a model could exploit such artifacts.
Statistical Artifacts: The Length Bias
A key finding of the study is the strong correlation between sentence length and target labels in several RSG tasks.
The boxplots illustrate that the "Number of words" is not uniformly distributed across classes. In many instances, the median length of a premise or hypothesis varies significantly depending on the label, providing a clear non-semantic signal that models can leverage to "cheat" on the task.
Key Results and Insights
The heuristic-based approach achieved an average RSG score of 0.468, outperforming RuGPT3-Small and remaining competitive with larger models like RuGPT3-Medium and RuBERT.
Specific task highlights include:
- RWSD (Winograd Schema): The majority class baseline matched the state-of-the-art performance (0.669 accuracy), suggesting that complex models were simply guessing the most frequent label rather than performing pronoun disambiguation.
- DaNetQA: Heuristics based on question structure (e.g., "Does it belong to...") achieved 0.642 accuracy, far exceeding the random baseline of 0.494.
- Data Leakage: The study found that class imbalances in the training set are often mirrored in the test set. A random weighted baseline scores 0.538, significantly higher than a purely random baseline (0.472), which artificially inflates model performance.
Conclusion and Recommendations
The success of these simple heuristics suggests that many top-performing models on the RSG leaderboard may be exploiting shallow statistical cues rather than demonstrating genuine linguistic reasoning. To improve the robustness of the benchmark, the authors recommend balancing test sets to eliminate data leakage, removing obvious length-based cues, and incorporating adversarial datasets specifically designed to trick models that rely on shallow heuristics.
Summary updated