TL;DR
This paper introduces Invertible Neural Networks (INNs) to natural language processing by framing morphological tasks—like inflection and analysis—as forward and inverse problems. By utilizing a single model with bi-directional training and categorical latent variables, the authors achieve highly competitive performance in both directions simultaneously, significantly reducing memory overhead while exploiting shared linguistic knowledge.
What Problem Does This Paper Address?
Many problems in natural language processing (NLP) are inherently dual. For example, morphological inflection takes a root word (lemma) and a set of grammatical tags to generate a modified "surface form" (e.g., cat + Plural $\rightarrow$ cats). Its inverse, morphological analysis, does the exact opposite (cats $\rightarrow$ cat + Plural).
Traditionally, NLP researchers train two entirely separate models to handle each direction. This approach not only doubles the parameters and memory requirements, but it also fails to exploit the shared underlying knowledge between the two tasks. Furthermore, inverse problems are frequently "ill-posed" or ambiguous: a single surface form might map to multiple valid lemmas or tag combinations depending on the context.
The authors address this gap by proposing a single, unified framework capable of learning both directions at once.
Methods
The authors adapt Invertible Neural Networks (INNs), originally designed for physical sciences and computer vision, to the discrete domain of text.
Key components of their approach include:
- Invertible Architecture: Built using invertible affine coupling layers, the network can be run forward ($x \rightarrow y$) and backward ($y \rightarrow x$) using the exact same model parameters.
- Bi-directional Training: The network is optimized at both ends simultaneously by alternating forward and inverse training iterations and accumulating the gradients.
- Categorical Latent Variables: To resolve the one-to-many ambiguity of ill-posed inverse problems, INNs introduce an extra latent variable $z$ to create a bijective mapping: $x \leftrightarrow [y, z]$. While vision models use continuous Gaussian distributions for $z$, text features are discrete. The authors instead utilize categorical latent variables approximated via the differentiable Gumbel-Softmax trick.
- Subword Embeddings: Words are split into subword units using Byte-Pair Encoding (BPE) and represented as the sum of pretrained 100-dimensional multilingual embeddings (BPEmb).
Datasets
The framework was evaluated on two standard morphological datasets across six typologically diverse languages:
- Celex: Used for French (FR), Finnish (FIN), and Romanian (RO).
- Wicentowski: Used for Turkish (TR), Tagalog (TL), and Irish (GA).
These languages span different morphological families, including agglutinative languages (where morphemes join together linearly, like Turkish) and fusional languages (where a single morpheme packs multiple grammatical meanings, like French).
Key Insights and Ideas
- Shared Parameters Work: The paper proves that a single set of weights can successfully encode both generation and analysis, challenging the standard NLP convention of separate modeling.
- The Power of Discrete Latents: Introducing categorical latent variables ($z$) via Gumbel-Softmax led to consistent improvements in lemmatization across all tested languages. Increasing the dimension of $z$ gave the network a larger, more flexible representation space to capture complex morphological traits.
- Implicit Learning: Fascinatingly, even when the network was optimized purely for the forward task (without explicit tag supervision), it implicitly began to learn simple morphological tag distributions, demonstrating the natural synergy of the invertible architecture.
Key Results
- Free Inverse Mapping: The single INN model achieved performance within 2% to 6% of a strong, state-of-the-art baseline designed solely for the forward task (Aharoni and Goldberg, 2017), while providing the inverse task entirely for free.
- Beating the Baseline: In Turkish lemmatization, the INN actually outperformed the reference model, achieving an exact match score of 99.54% compared to the baseline's 96.53%.
- Crucial Bi-directionality: The experiments showed that adding reverse training drastically boosted lemma prediction performance (the inverse problem) while causing only a negligible drop in the forward inflection task.
Limitations and Future Work
- Low Generation Diversity: When sampling surface forms using the latent variable $z$, the diversity of the generated words was quite low. The authors attribute this to either dominant tag combinations in the dataset or "posterior collapse," a common issue where the network learns to ignore the latent variable entirely.
- Vocabulary Constraints: Because the model relies on a nearest-neighbor search in a fixed embedding space to predict words, it cannot easily generate completely novel, unseen words out-of-the-box.
- Scaling to Complex Tasks: As an initial exploration of INNs in morphology, the model operates at the word level. The authors note that future work should explore scaling this architecture to sentence-level NLP problems, such as semantic parsing $\leftrightarrow$ text generation.
Summary updated