Morphological Verb-Aware Tibetan Language Model

Kuntharrgyal Khysru, Di Jin, Jianwu DangIEEE Access 2019 · OpenReview

Abstract

The Tibetan language model (TLM) is the key to Tibetan natural language processing. In this paper, we first observe that, different from widely used languages, Tibetan contains many morphological verbs that rarely appear in natural sentences but play a key role in accurate text prediction. This property is usually ignored by existing methods and makes traditional training strategies less effective in constructing accurate and robust TLMs. Hence, we propose a morphological verb-aware TLM by offline learning via a character frequency reweighting strategy and online tuning of discriminative weights conditioned on morphological verbs. However, because of the influence of morphological verbs on the tense and semantics of sentences, it is necessary to consider the morphological verbs in Tibetan. As a result, compared with state-of-the-art methods, our method not only reduces the perplexity but also improves the character error on tasks of the text prediction and automatic speech recognition (ASR).

Introduction

Building effective Natural Language Processing (NLP) tools for Tibetan is a significant challenge because it is a "low-resource" language. Unlike English or Mandarin, there is a lack of massive digitized datasets available to train modern AI models. This paper addresses a specific linguistic hurdle that has previously hindered the performance of Tibetan Language Models (TLMs): the complexity of morphological verbs.

In Tibetan, a single verb can take up to four different forms depending on the tense—future, present, past, and command. While these verbs are essential for understanding the meaning and timing of a sentence, many specific forms appear very rarely in natural text. Standard Recurrent Neural Network Language Models (RNNLMs) tend to ignore these "rare words," leading to high error rates in applications like Automatic Speech Recognition (ASR) and predictive texting.

Methodology

The researchers propose a two-pronged approach to make the TLM "aware" of these critical verbs, addressing the issue during both the training and inference phases.

1. Offline Learning via Character Frequency Reweighting (CFR)

Standard models typically group words into "classes" based on their frequency. Rare verbs often end up in low-priority buckets that the model does not emphasize. The proposed CFR strategy manually boosts the importance of these verbs by multiplying their frequency by a factor ($\beta = 3$) during the classification phase. This forces the model to treat these rare but semantically vital verbs as high-priority features during training.

2. Online Tuning of Discriminative Weights (TDW)

During the prediction phase (inference), the model employs a thresholding strategy. If the initial output suggests a morphological verb is likely, the model applies a secondary tuning step to "sharpen" the probability distribution. This helps the model commit more confidently to the correct verb form based on the specific context of the sentence, rather than defaulting to a more common but incorrect word.

Datasets and Experimental Setup

The study utilized two primary types of data to validate the model:

  • Text Data: A Small Tibetan Training dataset (STD) of 1.5 million tokens and a Large Tibetan Training dataset (LTD) of 21.3 million tokens, sourced from internet news covering politics, economics, and culture.
  • Audio Data: A speech corpus consisting of over 38,700 sentences recorded from 23 native speakers of the Lhasa dialect.
Proportion of morphological verbs across different Tibetan datasets.

As shown in the figure above, morphological verbs constitute a significant portion of the datasets (ranging from 14% to 25%), yet their individual forms often suffer from data sparsity, necessitating the specialized weighting strategies proposed in this research.

Key Results

The "verb-aware" model outperformed several state-of-the-art baselines, including traditional n-gram models and radical-based models, across multiple metrics:

  • Perplexity (PPL): The proposed model significantly improved the predictability of the language, reducing PPL from 62.9 (standard RNNLM) to 49.8 with the addition of TDW.
  • Character Error Rate (CER): In ASR tasks, the model reduced the error rate from 34.6% to 33.1%. While numerically small, this shift represents a meaningful improvement in the readability and accuracy of transcribed speech.
  • Subjective Accuracy: In a blind test conducted with five Tibetan language experts, the proposed model accurately represented sentence semantics in 47% of cases where the baseline model failed.

Conclusion

This research demonstrates that for morphologically rich languages like Tibetan, statistical frequency does not always equate to semantic importance. By integrating specific linguistic knowledge into the machine learning pipeline—specifically through the CFR and TDW strategies—the researchers created a model that is robust to the data sparsity that typically plagues low-resource languages. Future work will look toward generalizing these findings to other Tibetan dialects, such as Kham and Amdo, and scaling the approach to larger transformer-based architectures.

Summary updated

Abstract

The Tibetan language model (TLM) is the key to Tibetan natural language processing. In this paper, we first observe that, different from widely used languages, Tibetan contains many morphological verbs that rarely appear in natural sentences but play a key role in accurate text prediction. This property is usually ignored by existing methods and makes traditional training strategies less effective in constructing accurate and robust TLMs. Hence, we propose a morphological verb-aware TLM by offline learning via a character frequency reweighting strategy and online tuning of discriminative weights conditioned on morphological verbs. However, because of the influence of morphological verbs on the tense and semantics of sentences, it is necessary to consider the morphological verbs in Tibetan. As a result, compared with state-of-the-art methods, our method not only reduces the perplexity but also improves the character error on tasks of the text prediction and automatic speech recognition (ASR).
BibTeX
@inproceedings{khysru2019morphological,
  author = {Kuntharrgyal Khysru and Di Jin and Jianwu Dang},
  title = {{Morphological Verb-Aware Tibetan Language Model}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=YtaDLD9WJuI},
}
Figure