RUSSE: The First Workshop on Russian Semantic Similarity

Alexander Panchenko, Natalia V. Loukachevitch, Dmitry Ustalov, Denis Paperno, Christian Meyer, Natalia KonstantinovaarXiv 2018 · OpenReview

Abstract

The Russian Semantic Similarity Evaluation (RUSSE) shared task, held in conjunction with the Dialogue 2015 conference, represents the first systematic attempt to evaluate semantic similarity measures for the Russian language. Despite the maturity of this field for English and German, Russian's rich morphology and free word order have historically complicated such analysis. This paper provides an overview of the task, the four novel benchmark datasets introduced, and an analysis of 105 submissions from 19 teams. Our findings suggest that while supervised ensembles achieve the highest performance, unsupervised distributional models developed for English are remarkably effective for Russian as well.

The Challenge of Russian Semantics

In natural language processing (NLP), "semantic similarity" quantifies the relationship between words—distinguishing, for instance, the close bond between car and automobile from the lack of relation between car and cloud. While well-studied in English, Russian presents a "morphologically rich" landscape where a single root can manifest in dozens of forms, and word relationships are not always captured by position alone. Before RUSSE, researchers lacked a "gold standard" to compare methodologies, often adapting English-centric models in an ad-hoc manner.

Methodology and Shared Task

The authors organized the RUSSE shared task to bridge this gap, inviting participants to calculate similarity scores for nearly 15,000 word pairs. The 105 submissions from 19 teams generally fell into four categories:

  • Distributional Models: Utilizing context windows or syntactic structures (e.g., Word2vec, GloVe).
  • Knowledge-Based Measures: Leveraging linguistic ontologies such as Wiktionary and Wikipedia.
  • Supervised Models: Ensembles combining lexical, morphological, and orthographic features.
  • Network-Based Measures: Exploiting the structure of lexical graphs.

Benchmarking Russian Similarity

A core contribution of this work is the release of four novel datasets, providing a standardized framework for future research. These datasets cover human judgments, formal taxonomic relations, and cognitive associations.

Overview of the four novel benchmark datasets introduced for the RUSSE shared task.

As shown in the table above, the HJ (Human Judgments) dataset consists of 333 pairs translated from classic English benchmarks and scored via crowdsourcing. In contrast, the RT (Synonyms and Hypernyms), AE (Cognitive Associations), and AE2 (Cognitive Associations) datasets are structured as balanced classification tasks (50% related, 50% unrelated), where systems must distinguish between related and unrelated pairs.

Key Insights and Results

The evaluation utilized Spearman’s rank correlation for the HJ dataset and Average Precision (AVEP) for the classification tasks. The results yielded several significant insights:

  1. Robustness of Distributional Models: Models like Word2vec (Skip-gram and CBOW), originally designed for English, performed exceptionally well on Russian without requiring massive architectural changes.
  2. Taxonomy vs. Association: Systems were generally better at modeling natural "cognitive associations" (e.g., coffee and cup) than formal taxonomic relations (e.g., bird and cock).
  3. The Advantage of Ensembles: The highest scores were achieved by supervised ensembles. For example, the 5-rt-3 model (a Word2vec variant combined with synonym databases) led the HJ benchmark with a score of 0.7625.
  4. Unsupervised Performance: Despite the complexity of supervised models, a "pure" unsupervised Skip-gram model trained on a large raw corpus still ranked among the top 5 systems.
Precision-Recall curves for the top-performing models on the AE2 and RT benchmarks, showing high AUC values.

On the AE2 association benchmark, the top model (1-ae-1) achieved a near-perfect AVEP of 0.9849, while the best knowledge-based model for the RT benchmark (12-rt-3) reached 0.9589, as illustrated in the Precision-Recall curves above.

Conclusion and Future Directions

The RUSSE shared task successfully established a baseline for Russian semantic similarity. While global similarity measures are now well-benchmarked, the authors note that many words are polysemous (having multiple meanings). Future work should transition toward contextual semantic similarity, exploring how word relationships shift within specific sentences and further investigating how Russian's unique morphology can be better leveraged by next-generation models.

Summary updated

Abstract

The paper gives an overview of the Russian Semantic Similarity Evaluation (RUSSE) shared task held in conjunction with the Dialogue 2015 conference. There exist a lot of comparative studies on semantic similarity, yet no analysis of such measures was ever performed for the Russian language. Exploring this problem for the Russian language is even more interesting, because this language has features, such as rich morphology and free word order, which make it significantly different from English, German, and other well-studied languages. We attempt to bridge this gap by proposing a shared task on the semantic similarity of Russian nouns. Our key contribution is an evaluation methodology based on four novel benchmark datasets for the Russian language. Our analysis of the 105 submissions from 19 teams reveals that successful approaches for English, such as distributional and skip-gram models, are directly applicable to Russian as well. On the one hand, the best results in the contest were obtained by sophisticated supervised models that combine evidence from different sources. On the other hand, completely unsupervised approaches, such as a skip-gram model estimated on a large-scale corpus, were able score among the top 5 systems.
BibTeX
@misc{panchenko2018russe,
  author = {Alexander Panchenko and Natalia V. Loukachevitch and Dmitry Ustalov and Denis Paperno and Christian Meyer and Natalia Konstantinova},
  title = {{RUSSE: The First Workshop on Russian Semantic Similarity}},
  year = {2018},
  eprint = {1803.05820},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.05820},
}
Figure