Abstract
The paper describes the preparation and development of text collections within the framework of the MorphoRuEval-2017 shared task, an evaluation campaign designed to stimulate the development of automatic morphological processing technologies for Russian. The main challenge was to standardize four major Russian corpora—the Russian National Corpus (RNC), SynTagRus, OpenCorpora.org, and the General Internet-Corpus of Russian (GICR)—into a single Universal Dependencies (UD) CONLL-U format. These sources exhibited different tagsets, lemmatization rules, and technical architectures. The resulting collection includes approximately 3.6 million tokens across normative texts (news, literature) and informal discourse (social media, spoken data), available under a CC BY-NC-SA 3.0 license.
Introduction: Solving the "Tower of Babel" in Russian NLP
Morphological tagging—the identification of parts of speech (POS) and grammatical features like case, gender, and tense—is a foundational task in Natural Language Processing. Historically, Russian NLP has been fragmented. Different research groups developed independent corpora using unique tagsets and dictionary standards. This fragmentation made it impossible to conduct "apples-to-apples" comparisons of tagging algorithms; a high performance score on one dataset rarely translated to another.
The MorphoRuEval-2017 initiative addressed this problem by unifying these disparate resources into a consistent evaluation framework, providing a high-quality benchmark for the community.
Methodology: The Universal Dependencies Standard
The authors adopted the Universal Dependencies (UD) framework as their target standard. UD is an international initiative aimed at creating cross-linguistically consistent grammatical annotation. The standardization process was guided by "Manning’s Laws," which dictate that a morphological standard should be concise, internationally compatible, and easy for both humans and machines to process.
The conversion involved several complex technical hurdles:
- Normalization: Fixing "deficient" tagsets (e.g., handling plural-only nouns) and resolving conflicting grammatical interpretations.
- Algorithmic Discrimination: Developing logic to distinguish between homonymous categories, such as adverbs versus short-form adjectives.
- The "Predicative" Challenge: Handling words like mozhno ("one can"), which different linguistic schools categorize as adverbs, verbs, or a separate category. The authors eventually mapped these based on syntactic role to maintain UD compatibility.
The Unified Dataset
The project unified approximately 3.65 million annotated tokens from four primary sources:
- RNC Open (1.35M words): Manually disambiguated fiction, news, and spoken data.
- GICR (1M words): Informal language from blogs and social media.
- OpenCorpora.org (400k tokens): Crowdsourced data from Wikipedia and news.
- SynTagRus (900k tokens): A corpus traditionally used for syntactic research.
Additionally, 360 million words of unannotated plain text were provided to support unsupervised learning techniques.
Key Results: A Rosetta Stone for Russian POS
The primary output is the MorphoRuEval-2017 gold-standard dataset. A critical component of this work is the comprehensive mapping of POS tags across the four source systems and the UD standard. This mapping serves as a "Rosetta Stone" for the Russian NLP community.
Beyond POS mapping, the authors established standardized lemmatization rules (e.g., participles lemmatized to full nominative masculine singular adjectives), significantly reducing ambiguity for developers.
Conclusion
While the authors acknowledge that theoretical linguistic disputes remain, the MorphoRuEval-2017 dataset provides a permanent foundation for Russian NLP. By reconciling decades of divergent linguistic traditions into a single UD-compatible format, this work enables the training and evaluation of next-generation neural morphological taggers on a truly unified scale.
Summary updated