Abstract
In this paper, we utilize word embeddings focused on sub-word units for the Mongolian Phrase Break (PB) prediction task using a Long Short-Term Memory (LSTM) model. Mongolian is an agglutinative language where a single root can be followed by several suffixes to form millions of words. Existing Mongolian corpora are insufficient to build robust whole-word embeddings, leading to serious data sparsity. To solve this, we analyze sub-word units, encode their information into meaningful representations, and feed them into an LSTM to decode the optimal PB labels. Experimental results show that the proposed model significantly outperforms traditional Conditional Random Field (CRF) models using manual features, obtaining a 7.49% F-measure gain.
Introduction
In Text-to-Speech (TTS) systems, Phrase Break (PB) prediction is the critical first step in prosody modeling. Accurate pauses are essential for natural-sounding speech; incorrect placement results in robotic or confusing output. For Mongolian, this task is particularly challenging due to its agglutinative nature. A single root can combine with various suffixes to create millions of potential word forms, creating a massive data sparsity problem where even large corpora cannot cover all possible word forms.
The Challenge of Agglutinative Languages
Traditional PB prediction models rely heavily on Part-of-Speech (POS) tags, which are expensive and difficult to generate accurately for low-resource languages like Mongolian. Furthermore, the sheer variety of word forms makes standard word-level embeddings ineffective.
As shown in the figure above, Mongolian text involves complex relationships between the script, Latin transcription, and the necessary segmentation for processing. The researchers propose moving away from whole-word processing toward sub-word units to alleviate these issues.
Proposed Methodology
The authors developed a pipeline that decomposes Mongolian words into stems and suffixes. Specifically, they target suffixes attached via a Narrow Non-Break Space (NNBS), which are pervasive in the language.
- Segmentation Layer: Words like homun-u (of a person) are split into a stem (homun) and a suffix (-u).
- Sub-word Embeddings: A Skip-Gram model was trained on a 200-million-token unlabeled corpus to generate continuous vector representations for these units. This captures semantic and syntactic information—acting as a proxy for POS tags—without manual labeling.
- LSTM Architecture: These embeddings are fed into an LSTM network, which captures long-range dependencies across the sentence to decide if a break ("B") or non-break ("NB") should follow a token.
Experimental Results
The researchers found that splitting words into stems and suffixes significantly reduced the "Out of Vocabulary" problem. Suffixes often carry the syntactic weight that POS tags provide in other languages, allowing the model to learn syntactic rules automatically.
Key findings include:
- Optimal Dimensions: An embedding dimension of 150 was found to be the "sweet spot." Smaller dimensions lacked context, while larger ones (e.g., 300) introduced noise.
- Model Performance: The proposed LSTM model with sub-word embeddings (LEB) achieved an F-measure of 89.89%, outperforming the baseline CRF model (82.40%) by 7.49%.
- Sub-word Superiority: In all test scenarios, models using sub-word units consistently outperformed those using whole words or stems alone.
Conclusion
By leveraging sub-word embeddings and the sequential memory of LSTMs, this approach effectively overcomes the data sparsity inherent in agglutinative languages. This methodology provides a blueprint for improving NLP tasks in other low-resource, morphologically rich languages like Turkish, Japanese, or Korean. Future work may explore even more granular unsupervised morpheme segmentation.
Summary updated