A LSTM Approach with Sub-Word Embeddings for Mongolian Phrase Break Prediction

Rui Liu, Feilong Bao, Guanglai Gao, Hui Zhang, Yonghe WangCOLING 2018 · OpenReview

Abstract

In this paper, we utilize word embeddings focused on sub-word units for the Mongolian Phrase Break (PB) prediction task using a Long Short-Term Memory (LSTM) model. Mongolian is an agglutinative language where a single root can be followed by several suffixes to form millions of words. Existing Mongolian corpora are insufficient to build robust whole-word embeddings, leading to serious data sparsity. To solve this, we analyze sub-word units, encode their information into meaningful representations, and feed them into an LSTM to decode the optimal PB labels. Experimental results show that the proposed model significantly outperforms traditional Conditional Random Field (CRF) models using manual features, obtaining a 7.49% F-measure gain.

Introduction

In Text-to-Speech (TTS) systems, Phrase Break (PB) prediction is the critical first step in prosody modeling. Accurate pauses are essential for natural-sounding speech; incorrect placement results in robotic or confusing output. For Mongolian, this task is particularly challenging due to its agglutinative nature. A single root can combine with various suffixes to create millions of potential word forms, creating a massive data sparsity problem where even large corpora cannot cover all possible word forms.

The Challenge of Agglutinative Languages

Traditional PB prediction models rely heavily on Part-of-Speech (POS) tags, which are expensive and difficult to generate accurately for low-resource languages like Mongolian. Furthermore, the sheer variety of word forms makes standard word-level embeddings ineffective.

Example of Mongolian text segmentation and phrase break labeling

As shown in the figure above, Mongolian text involves complex relationships between the script, Latin transcription, and the necessary segmentation for processing. The researchers propose moving away from whole-word processing toward sub-word units to alleviate these issues.

Proposed Methodology

The authors developed a pipeline that decomposes Mongolian words into stems and suffixes. Specifically, they target suffixes attached via a Narrow Non-Break Space (NNBS), which are pervasive in the language.

  1. Segmentation Layer: Words like homun-u (of a person) are split into a stem (homun) and a suffix (-u).
  2. Sub-word Embeddings: A Skip-Gram model was trained on a 200-million-token unlabeled corpus to generate continuous vector representations for these units. This captures semantic and syntactic information—acting as a proxy for POS tags—without manual labeling.
  3. LSTM Architecture: These embeddings are fed into an LSTM network, which captures long-range dependencies across the sentence to decide if a break ("B") or non-break ("NB") should follow a token.
The proposed LSTM architecture for phrase break prediction

Experimental Results

The researchers found that splitting words into stems and suffixes significantly reduced the "Out of Vocabulary" problem. Suffixes often carry the syntactic weight that POS tags provide in other languages, allowing the model to learn syntactic rules automatically.

Key findings include:

  • Optimal Dimensions: An embedding dimension of 150 was found to be the "sweet spot." Smaller dimensions lacked context, while larger ones (e.g., 300) introduced noise.
  • Model Performance: The proposed LSTM model with sub-word embeddings (LEB) achieved an F-measure of 89.89%, outperforming the baseline CRF model (82.40%) by 7.49%.
  • Sub-word Superiority: In all test scenarios, models using sub-word units consistently outperformed those using whole words or stems alone.

Conclusion

By leveraging sub-word embeddings and the sequential memory of LSTMs, this approach effectively overcomes the data sparsity inherent in agglutinative languages. This methodology provides a blueprint for improving NLP tasks in other low-resource, morphologically rich languages like Turkish, Japanese, or Korean. Future work may explore even more granular unsupervised morpheme segmentation.

Summary updated

Abstract

In this paper, we first utilize the word embedding that focuses on sub-word units to the Mongolian Phrase Break (PB) prediction task by using Long-Short-Term-Memory (LSTM) model. Mongolian is an agglutinative language. Each root can be followed by several suffixes to form probably millions of words, but the existing Mongolian corpus is not enough to build a robust entire word embedding, thus it suffers a serious data sparse problem and brings a great difficulty for Mongolian PB prediction. To solve this problem, we look at sub-word units in Mongolian word, and encode their information to a meaningful representation, then fed it to LSTM to decode the best corresponding PB label. Experimental results show that the proposed model significantly outperforms traditional CRF model using manually features and obtains 7.49% F-Measure gain.
BibTeX
@inproceedings{liu2018lstm,
  author = {Rui Liu and Feilong Bao and Guanglai Gao and Hui Zhang and Yonghe Wang},
  title = {{A LSTM Approach with Sub-Word Embeddings for Mongolian Phrase Break Prediction}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=8FLDYXd28q},
}
Figure