Abstract
We incorporate morphological supervision into character language models (CLMs) via multitasking and show that this addition improves bits-per-character (BPC) performance across 24 languages, even when the morphology data and language modeling data are disjoint. Analyzing the CLMs shows that inflected words benefit more from explicitly modeling morphology than uninflected words, and that morphological supervision improves performance even as the amount of language modeling data grows. We then transfer morphological supervision across languages to improve language modeling performance in the low-resource setting.
Introduction
Character language models (CLMs) are theoretically capable of capturing subword structures, making them ideal for morphologically rich languages like Russian or Finnish. Unlike word-level models, CLMs bypass fixed vocabulary issues and have direct access to the internal structure of words. However, learning complex morphological rules—such as verb conjugations or noun cases—purely from raw text remains a significant challenge. This research explores whether explicitly multitasking CLMs with morphological tag prediction can enhance general language modeling performance without requiring massive, fully annotated datasets.
Methodology
The authors propose a Multitask Learning (MTL) framework using a stacked LSTM. The primary task is standard character-level language modeling (predicting the next character), while the auxiliary task is predicting morphological tags (e.g., tense, gender, number).
A key innovation is the projection of word-level morphological tags onto the first character of each word. This "early placement" strategy provides the model with morphological context before it begins decoding the subsequent characters of the word. The models were evaluated in two training regimes:
- Fully Supervised: Text and morphology labels originate from the same dataset.
- Distant Supervision: The language model is trained on a large raw text corpus (Multilingual Wikipedia Corpus), while morphology is learned from a separate, disjoint annotated dataset (Universal Dependencies).
Experimental Results
The MTL approach consistently outperformed standard LM baselines and even more complex hierarchical architectures across diverse languages.
As shown in Table 1, the MTL model improved bits-per-character (BPC) across all seven tested languages in the distant supervision setting. For instance, Czech performance improved from 2.013 to 1.938 BPC, and Russian from 1.745 to 1.645 BPC. Notably, this relatively simple MTL LSTM outperformed hierarchical models (HCLM-cache), suggesting that explicit linguistic supervision can be more effective than architectural complexity.
Analysis and Insights
The benefits of morphological supervision are not uniformly distributed. Analysis reveals that inflected words—those that change form to express grammatical categories—gain significantly more from the MTL objective than uninflected words.
Across 16 of the 24 languages studied, the average BPC improvement for inflected words was 31% greater than for uninflected words (see Figure 1(c)). This confirms that the model leverages morphological tags specifically to better predict complex word forms.
Furthermore, the study found that increasing the volume of raw training data does not make morphological supervision redundant. In fact, the performance gap between the MTL model and the baseline often increased with more data. The approach also proved effective for cross-lingual transfer; for example, augmenting Slovak training with Czech data resulted in a substantial 0.333 BPC improvement.
Limitations and Future Directions
While effective, the MTL approach is sensitive to the quantity of morphological data. Ablation studies indicated that using less than 25% of available morphology data could actually degrade performance relative to the baseline. Additionally, the benefits were primarily driven by the absolute size of the morphological dataset rather than the inherent morphological complexity of the language. Future research could investigate unsupervised morphology discovery to extend these benefits to truly low-resource languages lacking formal treebanks.
Summary updated