Text collections for evaluation of Russian morphological taggers

Olga Lyashevskaya, Victor Bocharov, Alexey Sorokin, Tatiana Shavrina, D. V. Granovsky, Svetlana AlexeevaJournal of Linguistics/Jazykovedný Casopis 2017 · DOI

Abstract

The paper describes the preparation and development of text collections within the framework of the MorphoRuEval-2017 shared task, an evaluation campaign designed to stimulate the development of automatic morphological processing technologies for Russian. The main challenge was to standardize four major Russian corpora—the Russian National Corpus (RNC), SynTagRus, OpenCorpora.org, and the General Internet-Corpus of Russian (GICR)—into a single Universal Dependencies (UD) CONLL-U format. These sources exhibited different tagsets, lemmatization rules, and technical architectures. The resulting collection includes approximately 3.6 million tokens across normative texts (news, literature) and informal discourse (social media, spoken data), available under a CC BY-NC-SA 3.0 license.

Introduction: Solving the "Tower of Babel" in Russian NLP

Morphological tagging—the identification of parts of speech (POS) and grammatical features like case, gender, and tense—is a foundational task in Natural Language Processing. Historically, Russian NLP has been fragmented. Different research groups developed independent corpora using unique tagsets and dictionary standards. This fragmentation made it impossible to conduct "apples-to-apples" comparisons of tagging algorithms; a high performance score on one dataset rarely translated to another.

The MorphoRuEval-2017 initiative addressed this problem by unifying these disparate resources into a consistent evaluation framework, providing a high-quality benchmark for the community.

Methodology: The Universal Dependencies Standard

The authors adopted the Universal Dependencies (UD) framework as their target standard. UD is an international initiative aimed at creating cross-linguistically consistent grammatical annotation. The standardization process was guided by "Manning’s Laws," which dictate that a morphological standard should be concise, internationally compatible, and easy for both humans and machines to process.

The conversion involved several complex technical hurdles:

  • Normalization: Fixing "deficient" tagsets (e.g., handling plural-only nouns) and resolving conflicting grammatical interpretations.
  • Algorithmic Discrimination: Developing logic to distinguish between homonymous categories, such as adverbs versus short-form adjectives.
  • The "Predicative" Challenge: Handling words like mozhno ("one can"), which different linguistic schools categorize as adverbs, verbs, or a separate category. The authors eventually mapped these based on syntactic role to maintain UD compatibility.

The Unified Dataset

The project unified approximately 3.65 million annotated tokens from four primary sources:

  1. RNC Open (1.35M words): Manually disambiguated fiction, news, and spoken data.
  2. GICR (1M words): Informal language from blogs and social media.
  3. OpenCorpora.org (400k tokens): Crowdsourced data from Wikipedia and news.
  4. SynTagRus (900k tokens): A corpus traditionally used for syntactic research.

Additionally, 360 million words of unannotated plain text were provided to support unsupervised learning techniques.

Key Results: A Rosetta Stone for Russian POS

The primary output is the MorphoRuEval-2017 gold-standard dataset. A critical component of this work is the comprehensive mapping of POS tags across the four source systems and the UD standard. This mapping serves as a "Rosetta Stone" for the Russian NLP community.

Table 1: Mapping of Part of Speech (POS) tags across source corpora and Universal Dependencies (Part 1)
Table 1: Mapping of Part of Speech (POS) tags across source corpora and Universal Dependencies (Part 2)

Beyond POS mapping, the authors established standardized lemmatization rules (e.g., participles lemmatized to full nominative masculine singular adjectives), significantly reducing ambiguity for developers.

Conclusion

While the authors acknowledge that theoretical linguistic disputes remain, the MorphoRuEval-2017 dataset provides a permanent foundation for Russian NLP. By reconciling decades of divergent linguistic traditions into a single UD-compatible format, this work enables the training and evaluation of next-generation neural morphological taggers on a truly unified scale.

Summary updated

Abstract

Abstract The paper describes the preparation and development of the text collections within the framework of MorphoRuEval-2017 shared task, an evaluation campaign designed to stimulate development of the automatic morphological processing technologies for Russian. The main challenge for the organizers was to standardize all available Russian corpora with the manually verified high-quality tagging to a single format (Universal Dependencies CONLL-U). The sources of the data were the disambiguated subcorpus of the Russian National Corpus, SynTagRus, OpenCorpora.org data and GICR corpus with the resolved homonymy, all exhibiting different tagsets, rules for lemmatization, pipeline architecture, technical solutions and error systematicity. The collections includes both normative texts (the news and modern literature) and more informal discourse (social media and spoken data), the texts are available under CC BY-NC-SA 3.0 license.
BibTeX
@article{lyashevskaya2017text,
  author = {Olga Lyashevskaya and Victor Bocharov and Alexey Sorokin and Tatiana Shavrina and D. V. Granovsky and Svetlana Alexeeva},
  title = {{Text collections for evaluation of Russian morphological taggers}},
  journal = {Journal of Linguistics/Jazykovedný Casopis},
  year = {2017},
  doi = {10.1515/jazcas-2017-0035},
}
Figure