Char-RNN for Word Stress Detection in East Slavic Languages

Ekaterina Chernyak, Maria Ponomareva, Kirill MilintsevichNAACL 2019

TL;DR

This paper introduces a character-level Recurrent Neural Network (RNN) approach for automatically detecting word stress in East Slavic languages. By creating new annotated datasets for Russian, Ukrainian, and Belarusian, they demonstrate that a "global" RNN model outperforms a "local" one and that cross-lingual training can significantly improve performance, especially for low-resource languages.

What Problem Does This Paper Address?

Accurately predicting word stress in languages like Russian, Ukrainian, and Belarusian is notoriously difficult. Unlike languages with fixed stress rules, stress in these East Slavic languages can shift unpredictably even within the same word's paradigm. This ambiguity poses challenges for applications like text-to-speech systems and resolving homonyms (words spelled the same but with different meanings and stress). Traditional rule-based systems struggle with exceptions and words lacking clear morphological markers, while dictionary-based approaches fail for out-of-vocabulary words. While deep learning has shown promise, there's a need for robust, language-agnostic methods, especially given the limited availability of annotated data for many languages. This paper aims to bridge this gap by exploring character-level RNNs for this task, focusing on resource-poor East Slavic languages and investigating the benefits of cross-lingual learning.

Methods

The core of the paper's methodology lies in applying character-level RNNs for word stress detection, framing it as a sequence labeling task. Each character in a word is assigned a label: 0 for no stress and 1 for stress.

Two main RNN architectures were compared:

  • Local Model: This model treats each character's stress prediction independently. It uses a bidirectional LSTM as a hidden layer and then applies a separate dense layer with a softmax activation for each character to predict its stress label. This is akin to traditional language modeling or Named Entity Recognition (NER) approaches.
  • Global Model: This model takes a more holistic approach. It encodes the entire input sequence (word and its context) into a single vector representation using a bidirectional LSTM. This representation is then decoded to generate the entire sequence of stress labels. This encoder-decoder architecture aims to capture dependencies across the whole word and its context more effectively.

Preprocessing involved tokenizing text, extracting left and right word contexts (last three characters), and removing stress markers from these contexts. The model then learns to predict stress based on these character sequences.

Datasets

The paper introduces new datasets for word stress detection in Russian, Ukrainian, and Belarusian. These datasets were created by:

  1. Leveraging Universal Dependencies (UD) treebanks: The authors used existing UD treebanks for the three languages.
  2. Manual Annotation via Crowdsourcing: They employed a crowdsourcing platform (Yandex.Toloka) to manually annotate word stress. The annotation process involved presenting annotators with word trigrams (a word and its immediate left and right context) and asking them to select the correct stress position by capitalizing a vowel in the word. Each task was solved by three annotators, and only tasks with unanimous agreement were kept.
  3. Data Size: The resulting datasets comprised approximately 20K trigrams for Russian, 10K for Ukrainian, and 3K for Belarusian. These sizes are noted as being proportional to the original UD corpora.

The paper also mentions using a standard dataset derived from the Russian National Corpus (RNC) for initial model selection and comparison.

Key Insights and Ideas

  • Character-level RNNs are effective: The study confirms that character-level RNNs are a viable and powerful approach for word stress detection, even for languages with complex stress patterns.
  • Global model outperforms local: The "global" RNN architecture, which encodes the entire sequence before decoding, significantly outperforms the "local" model that makes independent predictions for each character. This suggests that understanding the word as a whole, rather than just character by character, is crucial for accurate stress prediction.
  • Cross-lingual transfer learning is beneficial: Training models on one East Slavic language and testing on another yields promising results. Crucially, combining data from multiple languages for training generally improves accuracy compared to training on a single language, especially for lower-resource languages like Belarusian.
  • Context is king: The paper emphasizes the importance of word context (left and right words) for accurate stress prediction, which is why dictionary-based approaches are insufficient.

Key Results

  • Model Selection: The global model achieved higher accuracy than the local model on the RNC dataset, leading to its selection for further experiments.
  • Monolingual Performance: The proposed approach achieved high-quality results in monolingual settings for Russian, Ukrainian, and Belarusian, demonstrating its applicability across these languages. While accuracy was lower than on the standard RNC dataset (likely due to smaller training set sizes), it still showed strong performance.
  • Cross-lingual Performance (Setting 1): The Ukrainian dataset proved to be a good source for training models for both Russian and Belarusian. However, the Belarusian dataset, being the smallest, was less effective as a sole training source for other languages.
  • Cross-lingual Performance (Setting 2): Adding data from other languages to the training set significantly improved accuracy. For instance, adding the Russian training dataset to the Belarusian training set boosted accuracy by approximately 10% when testing on Belarusian.

Limitations and Future Work

  • Dataset Size: The newly created datasets, while valuable, are relatively small (especially for Belarusian), which impacts the performance compared to larger, established datasets.
  • Homograph Performance: The paper notes that word stress detection accuracy is generally lower for homographs, although the context-aware approach still helps in many cases.
  • Future Work: The authors suggest extending this work by:
  • Annotating new datasets for other languages with word stress phenomena.
  • Developing cross-lingual neural models using alternative architectures, such as transformers.

Summary updated

BibTeX
@inproceedings{chernyak2019char,
  author = {Ekaterina Chernyak and Maria Ponomareva and Kirill Milintsevich},
  title = {{Char-RNN for Word Stress Detection in East Slavic Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure