Abstract
We incorporate morphological supervision into character language models (CLMs) via multitasking and show that this addition improves bits-per-character (BPC) performance across 24 languages, even when the morphology data and language modeling data are disjoint. Analyzing the CLMs shows that inflected words benefit more from explicitly modeling morphology than uninflected words, and that morphological supervision improves performance even as the amount of language modeling data grows. We then transfer morphological supervision across languages to improve performance in the low-resource setting.
Introduction
Character language models (CLMs) are widely used due to their ability to handle out-of-vocabulary words and their lack of reliance on fixed vocabularies. However, they often struggle with morphology, particularly in morphologically rich languages like Czech or Finnish, where a single root can produce numerous inflected forms. Since many specific inflections appear rarely in training data, standard CLMs fail to capture the underlying structural regularities of word formation. This research addresses this gap by explicitly supervising CLMs with morphological information.
Methodology
The study utilizes a stacked LSTM architecture enhanced with a Multitask Learning (MTL) objective. While the model predicts the next character in a sequence, it is simultaneously tasked with predicting morphological features—such as tense, gender, or case—for the current word.
A critical architectural finding was the "first character" trick: supervising morphological tags on the first character of a word proved more effective than on the last. This forces the model to "plan" the grammatical structure before generating the subsequent characters. The total loss function is a weighted sum of the standard language modeling loss and auxiliary losses for each morphological feature. This approach allows the model to learn from disjoint datasets, utilizing gold-standard annotations from Universal Dependencies (UD v2.3) and larger text corpora from the Multilingual Wikipedia Corpus (MWC).
Results and Analysis
The MTL models achieved state-of-the-art results, outperforming previous benchmarks like HCLM-cache across all 24 languages tested. As shown in Table 1, the MTL approach consistently reduces bits-per-character (BPC) compared to standard language models.
A key insight is that morphology is not redundant with scale. Contrary to the intuition that larger datasets allow models to "figure out" morphology implicitly, the performance gain from morphological supervision actually increased as the training data grew (showing a strong correlation of $r=0.93$).
Furthermore, the model showed a 31% greater improvement on inflected words compared to uninflected ones. This confirms that the supervision specifically aids the model in navigating complex structural patterns.
In low-resource settings, cross-lingual transfer proved highly effective. For instance, using Czech morphological data to supervise a Slovak model resulted in a significant 0.333 BPC improvement over a baseline trained solely on Slovak data.
Conclusion
This work demonstrates that explicit morphological supervision significantly enhances character language modeling. By multitasking character prediction with morphological tagging, models can better capture sub-word regularities, even when labeled data is scarce or disjoint from the primary text corpus. While these results were established using LSTMs, the principles of early-word structural planning and morphological multitasking offer promising avenues for modern Transformer-based architectures.
Summary updated