Flow-Adapter Architecture for Unsupervised Machine Translation

Yihong Liu, Haris Jabbar, Hinrich Schütze

Abstract

Unsupervised Neural Machine Translation (NMT) typically relies on shared encoders to align different languages into a single latent space, a constraint that often fails to capture unique linguistic characteristics. This paper introduces the Flow-Adapter architecture, which employs Normalizing Flows to explicitly map language-specific representations into a shared, language-agnostic semantic space. This approach provides a flexible bridge between languages without requiring parallel data, effectively addressing the "copying problem" common in unsupervised models.

Problem Statement

Standard NMT models require massive parallel corpora, which are often unavailable for low-resource languages. Unsupervised NMT attempts to learn from monolingual data but frequently relies on a shared encoder-decoder framework. The authors argue that forcing different languages into the exact same latent space is a bottleneck because languages possess unique internal characteristics (vocabulary, pragmatics, etc.). Furthermore, traditional latent variable models like Variational Autoencoders (VAEs) are difficult to train in unsupervised settings due to restrictive distribution assumptions.

The Flow-Adapter Architecture

The core innovation is the Flow-Adapter. Instead of a forced shared space, the model allows each language to maintain its own latent representation ($z_x$ for source, $z_y$ for target). It then uses Normalizing Flows (NFs)—invertible deep generative models—to bridge these spaces through a language-agnostic semantic vector $\epsilon$.

The inference pipeline follows a structured path:

  1. Source Encoder: Maps the source sentence to a latent representation $z_x$.
  2. Source Flow: Transforms $z_x$ into the semantic vector $\epsilon$.
  3. Target Flow: Transforms $\epsilon$ into the target-language latent representation $z_y$.
  4. Target Decoder: Generates the final translation from $z_y$.
The Flow-Adapter inference pipeline showing the transformation from language-specific latent spaces to a shared semantic space.

Training Strategy

The model is trained using three primary objectives to ensure robust translation without parallel data:

  • Denoising Auto-Encoding (DAE): The model learns to reconstruct original sentences from noisy inputs.
  • Back-Translation (BT): A synthetic parallel corpus is generated by translating target sentences back to the source, ensuring cycle consistency.
  • Maximum Likelihood Estimation (MLE): This objective ensures the flows accurately model the distribution of sentence representations in the latent space.

Key Results and Insights

The Flow-Adapter model achieved competitive results against established unsupervised state-of-the-art (SOTA) models like XLM and MASS. On the Multi30K dataset, flow-based variants (3-scf and 3-glow) consistently outperformed baselines across all translation directions.

A critical finding involves the "copying problem." In unsupervised NMT with shared decoders, models often fail by simply outputting the input sentence. As shown in the table below, while a standard shared decoder baseline fails (BLEU near 0), the Flow-Adapter enables the shared decoder to achieve high performance by forcing the latent representation through a transformation into the target domain.

Comparison of shared and separate decoder settings on Multi30K, highlighting how Flow-Adapters mitigate the copying problem.

On WMT benchmarks, the model (using 5-scf) achieved a BLEU score of 35.77 for en-fr, outperforming the XLM baseline (33.40) and remaining competitive with MASS (37.50).

Discussion and Limitations

The Flow-Adapter architecture respects the unique properties of each language while finding common semantic ground. Its modularity allows for independent training of language flows, which could scale to massively multilingual systems. However, limitations remain:

  • Pre-training Integration: Currently, flows are added to existing models. Simultaneous pre-training might capture richer semantic information.
  • Language Complexity: Performance varies by language pair, suggesting that inherent linguistic similarity still influences translation quality.
  • Multilingual Scaling: Future work is needed to explore "zero-shot" translation between language pairs not explicitly seen together during training.

Summary updated

Abstract

In this work, we propose a flow-adapter architecture for unsupervised NMT. It leverages normalizing flows to explicitly model the distributions of sentence-level latent representations, which are subsequently used in conjunction with the attention mechanism for the translation task. The primary novelties of our model are: (a) capturing language-specific sentence representations separately for each language using normalizing flows and (b) using a simple transformation of these latent representations for translating from one language to another. This architecture allows for unsupervised training of each language independently. While there is prior work on latent variables for supervised MT, to the best of our knowledge, this is the first work that uses latent variables and normalizing flows for unsupervised MT. We obtain competitive results on several unsupervised MT benchmarks.
BibTeX
@inproceedings{liu2022flow,
  author = {Yihong Liu and Haris Jabbar and Hinrich Schütze},
  title = {{Flow-Adapter Architecture for Unsupervised Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.89},
}
Figure