Introduction
Neural Machine Translation (NMT) typically treats words as atomic units, a method that struggles when applied to morphologically rich languages (MRLs) like Turkish, Arabic, and Czech. In these languages, a single root can generate thousands of surface forms (e.g., "go," "goes," "going," "went"), leading to extreme data sparsity and a "vocabulary explosion."
The current standard solution, Byte-Pair Encoding (BPE), breaks words into subword units based on statistical frequency. While effective, BPE is "linguistically blind"—it segments words without regard for meaning or grammar, often creating semantically ambiguous fragments. Alternatively, character-level models avoid arbitrary segmentation but require very deep, computationally expensive architectures to capture long-range dependencies.
To bridge this gap, this paper introduces the Latent Morphology Model (LMM). LMM is a hierarchical latent variable model designed to understand the internal structure of words without relying on predefined linguistic rules, achieving state-of-the-art translation accuracy and superior generalization to unseen words.
Methodology: The Latent Morphology Model
The authors propose a Hierarchical Latent Variable Model based on the Variational Auto-Encoder (VAE) framework. Instead of predicting a target word in one shot, the model generates it character-by-character.
As illustrated in the architecture above, the generation process is conditioned on two distinct latent representations derived from the word-level hidden state:
- Continuous Latent Lemma ($z$): A dense vector intended to capture the core lexical meaning (semantics) of the word.
- Sparse Latent Features ($f$): A set of approximately discrete features intended to capture morphosyntactic functions (grammar, such as tense, number, or case).
The "Sparsity Trick": To make the discrete features ($f$) differentiable for neural network training, the authors employ a "stretched-and-rectified" Kumaraswamy distribution. This allows the model to sample values that are exactly 0 or 1 (simulating discrete grammatical categories) while still allowing gradients to flow during backpropagation. The model is trained using Variational Inference, optimizing a loss function that balances translation accuracy (Negative Log-Likelihood) with a sparsity-inducing regularizer that encourages the use of as few inflectional features as possible.
Experiments and Key Results
The model was evaluated on English translation into three languages with distinct morphological typologies: Arabic (Templatic), Czech (Fusional), and Turkish (Agglutinative). Training utilized low-resource TED Talks corpora, as well as a larger, multi-domain Turkish dataset to test generalization under high data sparsity.
- Superior Accuracy: LMM outperformed BPE, character-level, and standard hierarchical baselines across all three languages. In Arabic and Turkish, it achieved improvements of 0.51 and 0.30 BLEU points, respectively, over the best baselines.
- Robustness to Sparsity: In the multi-domain Turkish experiment, LMM's advantage grew significantly. It outperformed subword models by 0.82 BLEU and character-level models by a massive 2.54 BLEU.
- Unsupervised Morphology Discovery: Remarkably, the model learns morphological patterns entirely from raw text without a dictionary or morphological analyzer. It successfully disentangles semantics from grammar. Ablation studies showed that manually altering the latent features ($f$) forced the model to change a word's grammatical form while preserving its core meaning.
- Generalization over Memorization: Because LMM understands the process of word formation, it effectively handles "Out-of-Vocabulary" (OOV) words by combining a known lemma with appropriate inflectional features for the given context.
Limitations and Future Work
While highly effective, the LMM approach has limitations. Hierarchical models that generate text character-by-character are inherently slower than atomic word-level models, presenting a trade-off between translation accuracy and inference speed. Additionally, this study focused on Recurrent Neural Networks (RNNs). Future work will need to adapt this latent morphology approach to the now-dominant Transformer architecture. Finally, while the model is currently unsupervised, incorporating a small amount of gold-standard morphological data could potentially boost performance even further.
Summary updated