Long Context Benchmark for the Russian Language

Igor Churin, Murat Apishev, Maria Tikhonova, Denis Shevelev, Aydar Bulatov, Yuri Kuratov, Sergei Averkiev, Alena Fenogenova2025 · DOI

Abstract: Igor Churin, Murat Apishev, Maria Tikhonova, Denis Shevelev, Aydar Bulatov, Yuri Kuratov, Sergei Averkiev, Alena Fenogenova. Proceedings of the 6th Workshop on Computational Approaches to Discourse, Context and Document-Level Inferences (CODI 2025).

2025.

Introduction

As the context windows of Large Language Models (LLMs) have expanded from a few thousand tokens to over a million, the ability to evaluate these models has struggled to keep pace—particularly for non-English languages. While models can technically accept massive Russian inputs, researchers have lacked a standardized way to determine if these models are truly "understanding" the content or simply failing to process the noise.

To address this, researchers introduced LIBRA (Long Input Benchmark for Russian Analysis), the first comprehensive evaluation suite designed specifically for long-context Russian text. LIBRA provides a transparent methodology to measure retrieval, reasoning, and synthesis across documents up to 128k tokens.

The LIBRA Framework: Complexity and Fertility

The methodology behind LIBRA rests on two innovative pillars: complexity stratification and tokenizer standardization.

Complexity Stratification

Rather than treating all long-context tasks as equal, LIBRA divides its 18 datasets into four distinct groups based on the cognitive load required:

  • Group I: Simple "needle-in-a-haystack" retrieval tasks.
  • Group II: Single-hop Question Answering (QA) where the answer is explicitly stated in the text.
  • Group III: Multi-hop reasoning, requiring the model to connect multiple facts dispersed throughout the document.
  • Group IV: Complex synthesis, involving whole-document understanding and mathematical logic.
Overview of the LIBRA benchmark structure and complexity groups.

Token Fertility Standardization

Because different models use different tokenizers, a "100k token" limit represents varying amounts of actual text. To ensure a fair comparison, the authors calculated "tokenizer fertility"—the average number of tokens per word. By setting a standard fertility of 3.0, they ensured all models were tested on equivalent amounts of information across six length bins ranging from 4k to 128k tokens.

Dataset Composition

LIBRA consists of 18 datasets, 14 of which are entirely new and based on open Russian data sources like the Librusec library, ruSciBench, and Wikidata. The remaining four are high-quality translations of established English benchmarks. Notable new datasets include PasskeyWithLibrusec, which hides codes within literary text, and Matreshka, which tests a model's ability to track speakers in lengthy dialogues. Translated tasks include ruQuALITY (multiple-choice QA) and ruGSM100 (mathematical word problems).

Key Results and Insights

The researchers evaluated 17 popular LLMs, including proprietary models like GPT-4o and open-source families such as Llama, Mistral, and GLM. The results revealed several critical trends in long-context processing.

The Performance Gap

Proprietary models still hold a significant lead. GPT-4o dominated the leaderboard with an overall score of 65.4, followed by GLM4-9B-Chat at 50.0. While the Llama-3.1-8B family showed respectable performance (~43.0), accuracy for almost all models decayed as context length increased. For instance, GPT-4o’s accuracy dropped from 76.0% at the 4k limit to 53.6% at 128k.

Performance of various LLMs across different complexity groups in the LIBRA benchmark.

Retrieval vs. Reasoning

One of the most striking findings was that while many models excel at Group I (simple retrieval), performance plummets in Group IV. This suggests that current "long context" capabilities are often a retrieval trick rather than true reasoning over long sequences. A particularly difficult task was ruSciPassageCount, which requires counting unique paragraphs in a shuffled document; most models scored near zero, indicating a fundamental failure in global document structure awareness.

The Importance of SFT

Supervised Fine-Tuning (SFT) proved mandatory for long-context success. Raw pretrained models often "forgot" instructions or failed to maintain output formats when overwhelmed by long inputs, whereas SFT-tuned models remained significantly more robust.

Conclusion and Future Work

LIBRA highlights that processing long-context Russian text involves unique challenges, such as complex morphology and varying tokenization rates. While models are improving, true reasoning over massive documents remains an unsolved problem. Future research will need to address domain-specific performance (e.g., legal or medical), position bias (the "lost in the middle" phenomenon), and the potential for data leakage from classic Russian literature in training sets.

Summary updated

BibTeX
@misc{churin2025long,
  author = {Igor Churin and Murat Apishev and Maria Tikhonova and Denis Shevelev and Aydar Bulatov and Yuri Kuratov and Sergei Averkiev and Alena Fenogenova},
  title = {{Long Context Benchmark for the Russian Language}},
  year = {2025},
  doi = {10.18653/v1/2025.codi-1.1},
}
Figure