Abstract
Unsupervised Neural Machine Translation (NMT) typically relies on shared encoders to align different languages into a single latent space, a constraint that often fails to capture unique linguistic characteristics. This paper introduces the Flow-Adapter architecture, which employs Normalizing Flows to explicitly map language-specific representations into a shared, language-agnostic semantic space. This approach provides a flexible bridge between languages without requiring parallel data, effectively addressing the "copying problem" common in unsupervised models.
Problem Statement
Standard NMT models require massive parallel corpora, which are often unavailable for low-resource languages. Unsupervised NMT attempts to learn from monolingual data but frequently relies on a shared encoder-decoder framework. The authors argue that forcing different languages into the exact same latent space is a bottleneck because languages possess unique internal characteristics (vocabulary, pragmatics, etc.). Furthermore, traditional latent variable models like Variational Autoencoders (VAEs) are difficult to train in unsupervised settings due to restrictive distribution assumptions.
The Flow-Adapter Architecture
The core innovation is the Flow-Adapter. Instead of a forced shared space, the model allows each language to maintain its own latent representation ($z_x$ for source, $z_y$ for target). It then uses Normalizing Flows (NFs)—invertible deep generative models—to bridge these spaces through a language-agnostic semantic vector $\epsilon$.
The inference pipeline follows a structured path:
- Source Encoder: Maps the source sentence to a latent representation $z_x$.
- Source Flow: Transforms $z_x$ into the semantic vector $\epsilon$.
- Target Flow: Transforms $\epsilon$ into the target-language latent representation $z_y$.
- Target Decoder: Generates the final translation from $z_y$.
Training Strategy
The model is trained using three primary objectives to ensure robust translation without parallel data:
- Denoising Auto-Encoding (DAE): The model learns to reconstruct original sentences from noisy inputs.
- Back-Translation (BT): A synthetic parallel corpus is generated by translating target sentences back to the source, ensuring cycle consistency.
- Maximum Likelihood Estimation (MLE): This objective ensures the flows accurately model the distribution of sentence representations in the latent space.
Key Results and Insights
The Flow-Adapter model achieved competitive results against established unsupervised state-of-the-art (SOTA) models like XLM and MASS. On the Multi30K dataset, flow-based variants (3-scf and 3-glow) consistently outperformed baselines across all translation directions.
A critical finding involves the "copying problem." In unsupervised NMT with shared decoders, models often fail by simply outputting the input sentence. As shown in the table below, while a standard shared decoder baseline fails (BLEU near 0), the Flow-Adapter enables the shared decoder to achieve high performance by forcing the latent representation through a transformation into the target domain.
On WMT benchmarks, the model (using 5-scf) achieved a BLEU score of 35.77 for en-fr, outperforming the XLM baseline (33.40) and remaining competitive with MASS (37.50).
Discussion and Limitations
The Flow-Adapter architecture respects the unique properties of each language while finding common semantic ground. Its modularity allows for independent training of language flows, which could scale to massively multilingual systems. However, limitations remain:
- Pre-training Integration: Currently, flows are added to existing models. Simultaneous pre-training might capture richer semantic information.
- Language Complexity: Performance varies by language pair, suggesting that inherent linguistic similarity still influences translation quality.
- Multilingual Scaling: Future work is needed to explore "zero-shot" translation between language pairs not explicitly seen together during training.
Summary updated