Morphological Inflection Generation with Multi-space Variational Encoder-Decoders

Chunting Zhou, Graham NeubigCoNLL Shared Task 2017 · OpenReview

Abstract: Chunting Zhou, Graham Neubig. Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection.

  1. This paper presents the CMU submission to the SIGMORPHON 2017 shared task on morphological inflection. The authors employ a Multi-space Variational Encoder-Decoder (MSVED) that uses both continuous and discrete latent variables to model word roots and grammatical features, respectively, enabling the system to learn from both labeled examples and unlabeled Wikipedia text.

Introduction

In morphologically rich languages such as Turkish, Finnish, or Latin, a single root word (lemma) can manifest in hundreds of forms depending on grammatical context like tense, gender, number, and case. Automating this generation process is a fundamental challenge in Natural Language Processing (NLP). Traditionally, this requires extensive labeled datasets consisting of lemma + tags -> inflected form pairs. However, such data is expensive to produce. This study investigates whether semi-supervised learning—leveraging vast amounts of raw, unlabeled text—can improve a model's ability to generalize morphological patterns across 52 diverse languages.

Methodology: Multi-space Variational Encoder-Decoders

The authors propose a Multi-space Variational Encoder-Decoder (MSVED) framework, extending the standard Variational Autoencoder (VAE) for sequences. The "Multi-space" designation refers to the use of two distinct types of latent variables:

  1. Continuous Latent Variables ($z$): Designed to capture the semantic essence or the underlying lemma of the word.
  2. Discrete Latent Variables ($y$): Designed to represent specific morphological tags (e.g., POS=Verb, Tense=Past).
The architecture of the Unsupervised Variational Auto-encoder showing the multinomial sampling of discrete latent variables.

Technical Implementation

To overcome the non-differentiability of discrete choices during backpropagation, the authors utilize the Gumbel-Softmax trick. They also implement KL-Divergence Annealing to prevent "posterior collapse," where the model might otherwise ignore latent variables. Additionally, Decoder Input Dropout is employed to force the decoder to rely on the latent representations rather than becoming a simple language model. The final training objective combines reconstruction error from labeled data with an autoencoding loss on unlabeled data, where the model must infer the morphological tags of raw words.

Experimental Setup

The model was evaluated on the SIGMORPHON 2017 Shared Task 1 dataset, covering 52 languages. To facilitate semi-supervised learning, the authors incorporated Monolingual Wikipedia Data. Furthermore, they utilized Data Augmentation, expanding training sets by factors of 2 to 20 by generating additional word pairs sharing the same lemma.

Results and Analysis

The system achieved a competitive average accuracy of 86.21% across the 52-language test set. Performance varied significantly by language family and resource availability.

Development and Test accuracy across 52 languages, showing a wide range of performance.
  • High Performers: The model reached near-perfect accuracy in languages like Hindi (99.6%), Bengali (99.0%), and Welsh (99.0%).
  • Low Performers: Performance was lower in highly complex or lower-resource languages such as Latin (66.2%) and Icelandic (68.1%).
  • Ablation Study: Removing data augmentation resulted in significant accuracy drops (e.g., ~10% in Latin), indicating that synthetic data was often more beneficial than raw, unlabeled text.

Discussion: The Challenge of Semi-Supervised Learning

A key finding was that the semi-supervised model occasionally underperformed compared to supervised baselines. Analysis of Arabic and Persian revealed a distribution mismatch: the morphological tags in Wikipedia often differed drastically from the training data. For instance, if training data is dominated by verbs while Wikipedia text is primarily nouns, the unsupervised signal can confuse the model. This suggests that semi-supervised learning is highly sensitive to the relevance of the unlabeled corpus.

Conclusion

The MSVED approach demonstrates the potential of combining continuous and discrete latent spaces for morphological tasks. However, the quality and distributional alignment of unlabeled data remain critical hurdles. Future work will focus on better data selection strategies and modeling dependencies between different morphological tags to handle complex variations in languages like Latin and Icelandic.

Summary updated

BibTeX
@inproceedings{zhou2017morphological,
  author = {Chunting Zhou and Graham Neubig},
  title = {{Morphological Inflection Generation with Multi-space Variational Encoder-Decoders}},
  booktitle = {CoNLL Shared Task},
  year = {2017},
  url = {https://openreview.net/forum?id=0-epSP2Fgl},
}
Figure