A Latent Morphology Model for Open-Vocabulary Neural Machine Translation

Duygu Ataman, Wilker Aziz, Alexandra Birch

Introduction

Neural Machine Translation (NMT) typically treats words as atomic units, a method that struggles when applied to morphologically rich languages (MRLs) like Turkish, Arabic, and Czech. In these languages, a single root can generate thousands of surface forms (e.g., "go," "goes," "going," "went"), leading to extreme data sparsity and a "vocabulary explosion."

The current standard solution, Byte-Pair Encoding (BPE), breaks words into subword units based on statistical frequency. While effective, BPE is "linguistically blind"—it segments words without regard for meaning or grammar, often creating semantically ambiguous fragments. Alternatively, character-level models avoid arbitrary segmentation but require very deep, computationally expensive architectures to capture long-range dependencies.

To bridge this gap, this paper introduces the Latent Morphology Model (LMM). LMM is a hierarchical latent variable model designed to understand the internal structure of words without relying on predefined linguistic rules, achieving state-of-the-art translation accuracy and superior generalization to unseen words.

Methodology: The Latent Morphology Model

The authors propose a Hierarchical Latent Variable Model based on the Variational Auto-Encoder (VAE) framework. Instead of predicting a target word in one shot, the model generates it character-by-character.

Architecture of the Latent Morphology Model

As illustrated in the architecture above, the generation process is conditioned on two distinct latent representations derived from the word-level hidden state:

  1. Continuous Latent Lemma ($z$): A dense vector intended to capture the core lexical meaning (semantics) of the word.
  2. Sparse Latent Features ($f$): A set of approximately discrete features intended to capture morphosyntactic functions (grammar, such as tense, number, or case).

The "Sparsity Trick": To make the discrete features ($f$) differentiable for neural network training, the authors employ a "stretched-and-rectified" Kumaraswamy distribution. This allows the model to sample values that are exactly 0 or 1 (simulating discrete grammatical categories) while still allowing gradients to flow during backpropagation. The model is trained using Variational Inference, optimizing a loss function that balances translation accuracy (Negative Log-Likelihood) with a sparsity-inducing regularizer that encourages the use of as few inflectional features as possible.

Experiments and Key Results

The model was evaluated on English translation into three languages with distinct morphological typologies: Arabic (Templatic), Czech (Fusional), and Turkish (Agglutinative). Training utilized low-resource TED Talks corpora, as well as a larger, multi-domain Turkish dataset to test generalization under high data sparsity.

  • Superior Accuracy: LMM outperformed BPE, character-level, and standard hierarchical baselines across all three languages. In Arabic and Turkish, it achieved improvements of 0.51 and 0.30 BLEU points, respectively, over the best baselines.
  • Robustness to Sparsity: In the multi-domain Turkish experiment, LMM's advantage grew significantly. It outperformed subword models by 0.82 BLEU and character-level models by a massive 2.54 BLEU.
  • Unsupervised Morphology Discovery: Remarkably, the model learns morphological patterns entirely from raw text without a dictionary or morphological analyzer. It successfully disentangles semantics from grammar. Ablation studies showed that manually altering the latent features ($f$) forced the model to change a word's grammatical form while preserving its core meaning.
  • Generalization over Memorization: Because LMM understands the process of word formation, it effectively handles "Out-of-Vocabulary" (OOV) words by combining a known lemma with appropriate inflectional features for the given context.

Limitations and Future Work

While highly effective, the LMM approach has limitations. Hierarchical models that generate text character-by-character are inherently slower than atomic word-level models, presenting a trade-off between translation accuracy and inference speed. Additionally, this study focused on Recurrent Neural Networks (RNNs). Future work will need to adapt this latent morphology approach to the now-dominant Transformer architecture. Finally, while the model is currently unsupervised, incorporating a small amount of gold-standard morphological data could potentially boost performance even further.

Summary updated

Abstract

Translation into morphologically-rich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic. This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters. The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task. The latter learns directly from translation data but requires rather deep architectures. In this paper, we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inflection. Our model generates words one character at a time by composing two latent representations: a continuous one, aimed at capturing the lexical semantics, and a set of (approximately) discrete features, aimed at capturing the morphosyntactic function, which are shared among different surface forms. Our model achieves better accuracy in translation into three morphologically-rich languages than conventional open-vocabulary NMT methods, while also demonstrating a better generalization capacity under low to mid-resource settings.
BibTeX
@inproceedings{ataman2020latent,
  author = {Duygu Ataman and Wilker Aziz and Alexandra Birch},
  title = {{A Latent Morphology Model for Open-Vocabulary Neural Machine Translation}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=PkxzGirA64z},
}
Figure