A Latent Morphology Model for Open-Vocabulary Neural Machine Translation

Duygu Ataman, Wilker Aziz, Alexandra Birch

Abstract

Translation into morphologically-rich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic. This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters. The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task. The latter learns directly from translation data but requires rather deep architectures. In this paper, we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inflection. Our model generates words one character at a time by composing two latent representations: a continuous one, aimed at capturing the lexical semantics, and a set of (approximately) discrete features, aimed at capturing the morphosyntactic function, which are shared among different surface forms. Our model achieves better accuracy in translation into three morphologically-rich languages than conventional open-vocabulary NMT methods, while also demonstrating a better generalization capacity under low to mid-resource settings.

Introduction: The Challenge of Morphological Richness

Translating into languages like Turkish, Arabic, or Czech presents a significant hurdle for standard Neural Machine Translation (NMT) systems. These "morphologically rich" languages allow a single root word to take on thousands of distinct surface forms depending on its syntactic role (e.g., tense, number, case). Standard approaches typically fall into two categories:

  1. Subword Units (BPE): Statistical segmentation into frequent chunks. While effective, it ignores linguistic structure and can create semantically ambiguous fragments.
  2. Character-level Models: Generating text one character at a time. While flexible, these models require deep architectures to capture long-range dependencies and grammatical rules.

The core issue is "lexical sparsity": models struggle to predict word forms they haven't encountered in training, even if the root lemma is familiar.

The Latent Morphology Model (LMM)

The authors propose a Latent Morphology Model (LMM) that treats word formation as a stochastic process. Instead of predicting a word directly, the model decomposes it into two latent variables:

  • The Lemma ($z$): A continuous vector capturing the core lexical meaning.
  • Inflectional Features ($f$): A set of sparse, approximately discrete features representing morphosyntactic functions.
The LMM architecture: A word-level RNN produces a hidden state, which is then decomposed into a latent lemma (z) and inflectional features (F) to drive a character-level decoder.

To maintain differentiability during training, the authors utilize the Kumaraswamy distribution for the inflectional features. This "stretch-and-rectify" technique allows the model to sample features that are mostly 0 or 1, enabling the unsupervised discovery of grammatical categories without explicit labels. The final word is generated character-by-character by a decoder primed with the combination of these latent representations.

Experimental Setup

The model was evaluated on English translation into three morphologically diverse languages:

  • Arabic (Templatic morphology)
  • Czech (Fusional morphology)
  • Turkish (Agglutinative morphology)

Experiments were conducted using the TED Talks corpora for low-resource scenarios and a larger multi-domain Turkish corpus to assess scalability.

Key Insights and Results

The LMM consistently outperformed standard BPE and character-level baselines:

  • Performance Gains: In low-resource settings, the model achieved improvements of 0.51 BLEU (Arabic) and 0.30 BLEU (Turkish). In the larger multi-domain Turkish task, it showed a significant jump of 0.82 BLEU over subword models and 2.54 BLEU over character-level decoders.
  • Inductive Bias and Compression: By forcing a separation between "meaning" and "grammar," the model mimics human linguistic processes. Interestingly, compressing the latent representations (e.g., from 512 to 100 dimensions) improved accuracy, suggesting that tighter bottlenecks force the model to learn robust, generalized patterns.
  • Unsupervised Grammar Discovery: Despite no explicit supervision, the model learned to use its latent features to represent concepts like plurality or tense, effectively performing unsupervised morphological analysis.

Conclusion

The Latent Morphology Model offers a linguistically-motivated alternative to purely statistical subword methods. By explicitly modeling the hierarchical nature of word formation, it achieves superior generalization, particularly for unseen word forms in morphologically-rich target languages. Future work may explore the model's applicability to "isolating" languages like Mandarin and further optimize the decoding complexity inherent in character-level generation.

Summary updated

Abstract

Translation into morphologically-rich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic. This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters. The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task. The latter learns directly from translation data but requires rather deep architectures. In this paper, we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inflection. Our model generates words one character at a time by composing two latent representations: a continuous one, aimed at capturing the lexical semantics, and a set of (approximately) discrete features, aimed at capturing the morphosyntactic function, which are shared among different surface forms. Our model achieves better accuracy in translation into three morphologically-rich languages than conventional open-vocabulary NMT methods, while also demonstrating a better generalization capacity under low to mid-resource settings.
BibTeX
@inproceedings{ataman2020latent,
  author = {Duygu Ataman and Wilker Aziz and Alexandra Birch},
  title = {{A Latent Morphology Model for Open-Vocabulary Neural Machine Translation}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=PkxzGirA64z},
}
Figure