Towards the Data-driven System for Rhetorical Parsing of Russian Texts

Elena Chistova, Maria Kobozeva, Dina Pisarevskaya, Artem Shelmanov, Ivan Smirnov, Svetlana Toldova

TL;DR

This paper presents the first data-driven system for rhetorical parsing of Russian texts, leveraging the newly released Ru-RSTreebank corpus. The authors demonstrate that an ensemble of CatBoost and linear SVM models, with a strong emphasis on discourse connectives as features, achieves the best performance in classifying rhetorical relations.

What Problem Does This Paper Address?

Understanding the rhetorical structure of a text – how different parts of a text relate to each other to build a coherent argument or narrative – is crucial for many Natural Language Processing (NLP) tasks, such as text summarization, question answering, and machine translation. Rhetorical Structure Theory (RST) provides a framework for this, representing texts as hierarchical trees where units of text (Discourse Units or DUs) are connected by specific rhetorical relations (e.g., "Cause," "Contrast," "Elaboration"). While significant progress has been made in English, data-driven rhetorical parsing for other languages, particularly Russian, has been limited due to the lack of annotated corpora. This paper addresses this gap by developing and evaluating the first data-driven rhetorical parser for Russian, using the Ru-RSTreebank corpus.

Methods

The authors propose a pipeline for automated discourse parsing that involves five subtasks: sentence segmentation, relation prediction, discourse tree construction, nuclear-satellite classification, and labeling relations between DUs. For the core task of rhetorical relation classification, they experiment with several supervised learning algorithms, including logistic regression, linear SVM, and gradient boosting (LightGBM and CatBoost).

A key aspect of their methodology is feature engineering. They combine various types of features:

  • Lexical features: These include occurrences of discourse connectives (cue phrases) from a manually curated list, TF-IDF vectors of words, and cosine similarity measures.
  • Quantitative features: Basic text statistics like word count, average word length, and capitalization.
  • Morphological features: Vector counts of morphological characteristics and part-of-speech tags.
  • Semantic features: Averaged word embeddings of DUs.

To handle the high dimensionality and sparsity of features, they employ feature selection using L1-regularized logistic regression. They also explore ensemble methods, specifically soft-voting ensembles combining linear classifiers with gradient boosting models.

Datasets

The primary dataset used is the Ru-RSTreebank corpus. This is the first Russian corpus annotated within the RST framework. It contains 179 texts from genres like news, news analytics, popular science, and research articles in linguistics and computer science, totaling over 200,000 tokens. The corpus provides annotations for EDU segmentation, discourse unit nuclearity, types of discourse relations, and rhetorical tree construction. The authors also mention creating a lexicon of discourse connectives based on this corpus.

Key Insights and Ideas

  • Discourse connectives are king: The most surprising and important finding is the overwhelming importance of discourse connectives (cue phrases) for rhetorical relation classification. These words and phrases act as strong signals for the relationship between text segments.
  • Ensembles shine: Ensemble methods, particularly combining a powerful gradient boosting model (CatBoost) with a simpler linear model (linear SVM), yield the best performance, suggesting that different models capture complementary information.
  • Feature selection is crucial: For high-dimensional feature spaces, feature selection is not just about reducing computational cost but also about improving model robustness and performance by focusing on the most informative signals.
  • Ru-RSTreebank is a valuable resource: The creation and release of the Ru-RSTreebank corpus unlock the potential for applying machine learning techniques to Russian discourse parsing, paving the way for future research.

Key Results

  • The ensemble of CatBoost with selected features and a linear SVM model achieved the best performance for rhetorical relation classification, with a macro F1 score of 54.67±0.38.
  • The CatBoost model alone outperformed linear SVM and logistic regression for nuclear-satellite classification, achieving a macro F1 of 67.79±0.57.
  • Feature importance analysis revealed that lexical features, particularly those related to discourse connectives, were the most informative. Out of 3,624 features, CatBoost selected 2,054, with a significant portion being related to cue phrases.

Limitations and Future Work

  • Relation Exclusions: The authors excluded "Elaboration" and "Joint" relations due to their high frequency and lower informativeness, and "Same-unit" for annotation utility reasons. This means their results are based on a subset of all possible relations.
  • Skewed Distribution: The dataset exhibits a skewed distribution of classes, which is a common challenge in classification tasks.
  • Future Work: The authors plan to:
  • Apply an extended version of their discourse connectives lexicon for relation classification.
  • Implement more complex deep learning methods to further improve performance.
  • Continue developing the data-driven system for rhetorical parsing of Russian texts.

Summary updated

Abstract

Results of the first experimental evaluation of machine learning models trained on Ru-RSTreebank – first Russian corpus annotated within RST framework – are presented. Various lexical, quantitative, morphological, and semantic features were used. In rhetorical relation classification, ensemble of CatBoost model with selected features and a linear SVM model provides the best score (macro F1 = 54.67 ± 0.38). We discover that most of the important features for rhetorical relation classification are related to discourse connectives derived from the connectives lexicon for Russian and from other sources.
BibTeX
@inproceedings{chistova2019towards,
  author = {Elena Chistova and Maria Kobozeva and Dina Pisarevskaya and Artem Shelmanov and Ivan Smirnov and Svetlana Toldova},
  title = {{Towards the Data-driven System for Rhetorical Parsing of Russian Texts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure