Abstract
The Russian Semantic Similarity Evaluation (RUSSE) shared task, held in conjunction with the Dialogue 2015 conference, represents the first systematic attempt to evaluate semantic similarity measures for the Russian language. Despite the maturity of this field for English and German, Russian's rich morphology and free word order have historically complicated such analysis. This paper provides an overview of the task, the four novel benchmark datasets introduced, and an analysis of 105 submissions from 19 teams. Our findings suggest that while supervised ensembles achieve the highest performance, unsupervised distributional models developed for English are remarkably effective for Russian as well.
The Challenge of Russian Semantics
In natural language processing (NLP), "semantic similarity" quantifies the relationship between words—distinguishing, for instance, the close bond between car and automobile from the lack of relation between car and cloud. While well-studied in English, Russian presents a "morphologically rich" landscape where a single root can manifest in dozens of forms, and word relationships are not always captured by position alone. Before RUSSE, researchers lacked a "gold standard" to compare methodologies, often adapting English-centric models in an ad-hoc manner.
Methodology and Shared Task
The authors organized the RUSSE shared task to bridge this gap, inviting participants to calculate similarity scores for nearly 15,000 word pairs. The 105 submissions from 19 teams generally fell into four categories:
- Distributional Models: Utilizing context windows or syntactic structures (e.g., Word2vec, GloVe).
- Knowledge-Based Measures: Leveraging linguistic ontologies such as Wiktionary and Wikipedia.
- Supervised Models: Ensembles combining lexical, morphological, and orthographic features.
- Network-Based Measures: Exploiting the structure of lexical graphs.
Benchmarking Russian Similarity
A core contribution of this work is the release of four novel datasets, providing a standardized framework for future research. These datasets cover human judgments, formal taxonomic relations, and cognitive associations.
As shown in the table above, the HJ (Human Judgments) dataset consists of 333 pairs translated from classic English benchmarks and scored via crowdsourcing. In contrast, the RT (Synonyms and Hypernyms), AE (Cognitive Associations), and AE2 (Cognitive Associations) datasets are structured as balanced classification tasks (50% related, 50% unrelated), where systems must distinguish between related and unrelated pairs.
Key Insights and Results
The evaluation utilized Spearman’s rank correlation for the HJ dataset and Average Precision (AVEP) for the classification tasks. The results yielded several significant insights:
- Robustness of Distributional Models: Models like Word2vec (Skip-gram and CBOW), originally designed for English, performed exceptionally well on Russian without requiring massive architectural changes.
- Taxonomy vs. Association: Systems were generally better at modeling natural "cognitive associations" (e.g., coffee and cup) than formal taxonomic relations (e.g., bird and cock).
- The Advantage of Ensembles: The highest scores were achieved by supervised ensembles. For example, the
5-rt-3model (a Word2vec variant combined with synonym databases) led the HJ benchmark with a score of 0.7625. - Unsupervised Performance: Despite the complexity of supervised models, a "pure" unsupervised Skip-gram model trained on a large raw corpus still ranked among the top 5 systems.
On the AE2 association benchmark, the top model (1-ae-1) achieved a near-perfect AVEP of 0.9849, while the best knowledge-based model for the RT benchmark (12-rt-3) reached 0.9589, as illustrated in the Precision-Recall curves above.
Conclusion and Future Directions
The RUSSE shared task successfully established a baseline for Russian semantic similarity. While global similarity measures are now well-benchmarked, the authors note that many words are polysemous (having multiple meanings). Future work should transition toward contextual semantic similarity, exploring how word relationships shift within specific sentences and further investigating how Russian's unique morphology can be better leveraged by next-generation models.
Summary updated