Two Birds with One Stone: Investigating Invertible Neural Networks for Inverse Problems in Morphology

Gözde Gül Sahin, Iryna Gurevych

Introduction

In natural language processing (NLP), many tasks exist as symmetric pairs. Morphological inflection—generating a surface form from a root (lemma) and grammatical tags—is the inverse of morphological analysis, which recovers those roots and tags from a given word. Traditionally, these tasks are treated as separate problems requiring distinct models. This approach is not only computationally inefficient but also fails to exploit the inherent shared knowledge of a language's rules.

This paper investigates the application of Invertible Neural Networks (INNs) to these morphological "inverse problems." By utilizing a single set of model parameters to perform both analysis and generation, the researchers demonstrate a more efficient and theoretically sound approach to linguistic modeling.

Methodology: Invertible Neural Networks

The core innovation lies in the use of INNs, an architecture designed to be mathematically reversible. Unlike standard neural networks that map inputs to outputs in a one-way fashion, INNs learn a bijective (one-to-one and onto) mapping.

The Bijective Mapping

To account for the "ill-posed" nature of linguistic tasks—where a single input might correspond to multiple valid outputs (e.g., homonyms or multiple inflectional forms)—the authors introduce a latent variable $z$. The network learns the mapping: $$x \leftrightarrow [y, z]$$ In this framework, $x$ represents the input (such as a lemma and its tags), $y$ is the primary output (the surface form), and $z$ captures the remaining uncertainty or stylistic variation.

Handling Discrete Language Data

INNs typically operate on continuous data. To adapt them for morphology, the authors employ Byte-Pair-Encoding (BPE) embeddings for word representation and the Gumbel-Softmax trick for categorical morphological tags. This allows the model to maintain differentiability while processing discrete linguistic units.

Task Configurations

The researchers evaluated the INN framework across two primary configurations, as illustrated below.

The INN framework applied to two task pairs: Morphological Inflection/Analysis (left) and Lemmatization/Surface Form Sampling (right).
  1. Morphological Inflection and Analysis: The model maps a lemma and its tags to a surface form and a latent variable. In the reverse direction, it recovers the lemma and tags from the surface form.
  2. Lemmatization and Surface Form Sampling: The model maps a surface form to its lemma and a latent variable. Reversing this allows the model to "sample" various possible surface forms for a single lemma by varying the latent variable $z$.

Experimental Insights

The model was tested on six languages representing different morphological typologies, including agglutinative languages (Turkish, Finnish, Tagalog) and fusional languages (French, Romanian, Irish).

  • Shared Knowledge: Training on both directions simultaneously improved performance on the inverse task (analysis) with minimal impact on the forward task (inflection).
  • Latent Variable Clustering: Interestingly, the latent variable $z$ began to implicitly cluster morphological information even without explicit supervision, effectively acting as a "tag catcher."
  • Typological Performance: The model excelled in agglutinative languages where the relationship between word components and meanings is more transparent. In fusional languages, the mapping is more complex, presenting a greater challenge for the bijective architecture.

Conclusion

Invertible Neural Networks provide a powerful framework for handling bidirectional tasks in NLP. By treating morphology as an inverse problem, a single model can achieve competitive performance across multiple tasks while maintaining a unified representation of linguistic rules. Future work will explore scaling this framework to more complex structural transformations, such as semantic parsing and machine translation.

Summary updated

Abstract

Most problems in natural language processing can be approximated as inverse problems such as analysis and generation at variety of levels from morphological (e.g., cat+Plural <-> cats) to semantic (e.g., (call + 1 2) <-> "Calculate one plus two."). Although the tasks in both directions are closely related, general approach in the field has been to design separate models specific for each task. However, having one shared model for both tasks, would help the researchers exploit the common knowledge among these problems with reduced time and memory requirements. We investigate a specific class of neural networks, called Invertible Neural Networks (INNs) (Ardizzone et al. 2019) that enable simultaneous optimization in both directions, hence allow addressing of inverse problems via a single model. In this study, we investigate INNs on morphological problems casted as inverse problems. We apply INNs to various morphological tasks with varying ambiguity and show that they provide competitive performance in both directions. We show that they are able to recover the morphological input parameters, i.e., predicting the lemma (e.g., cat) or the morphological tags (e.g., Plural) when run in the reverse direction, without any significant performance drop in the forward direction, i.e., predicting the surface form (e.g., cats).
BibTeX
@inproceedings{sahin2020two,
  author = {Gözde Gül Sahin and Iryna Gurevych},
  title = {{Two Birds with One Stone: Investigating Invertible Neural Networks for Inverse Problems in Morphology}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6286},
}
Figure