Abstract
Unsupervised Neural Machine Translation (UNMT) faces a significant "cold start" problem due to the lack of explicit cross-lingual anchors. This paper proposes leveraging Universal Grammar—the linguistic theory that human languages share common structural building blocks—to bridge this gap. By introducing training objectives that mask and predict syntactic constituents and utilize back-translated context, the authors demonstrate substantial improvements in translation quality across multiple language pairs without the need for parallel data.
Introduction
Standard Machine Translation systems are notoriously data-hungry, requiring millions of parallel sentence pairs to achieve high performance. While Unsupervised NMT (UNMT) aims to mitigate this by training on monolingual data, it often struggles with initial alignment. The authors argue that while surface-level words differ, underlying grammatical structures like Noun Phrases (NP) and Verb Phrases (VP) are remarkably consistent across languages. By treating these shared structures as a "universal anchor," the model can learn to align languages in a shared latent space.
Methodology
To inject syntactic knowledge into the translation model, the authors propose three primary training objectives:
- CONSTMLM (Constituent Masked Language Modeling): Unlike standard BERT-style masking of random tokens, this method identifies and masks entire syntactic constituents. The model is tasked with predicting the missing words and their corresponding syntactic category (e.g., NP, VP). This forces the encoder to recognize structural patterns common to both source and target languages.
- BTLM (Back-translation Language Modeling): Masking large spans of text can make recovery nearly impossible. BTLM provides the model with a translated version of the sentence (back-translation) as a cross-lingual hint, enabling the recovery of long masked spans and promoting bilingual alignment.
- CONSTBTLM: A hybrid approach that combines constituent masking with cross-lingual context.
These objectives were implemented in both encoder-only and encoder-decoder configurations. The authors found that the encoder-decoder setup was particularly effective, as the decoder's generation task complements the encoder's structural recognition.
Experimental Setup
The approach was evaluated on standard benchmarks including WMT’14 (En-Fr), WMT’16 (En-De, En-Ro), and WMT’20 (En-Zh). Monolingual data was sourced from News Crawl and Wikipedia. To generate the necessary syntactic labels for training, the researchers utilized the Berkeley Neural Parser trained on established treebanks like the Penn Treebank.
Key Results
The proposed methods consistently outperformed the strong XLM baseline:
- Performance Gains: On English-French (En-Fr), the
CONSTBTLM enc-decmodel achieved a BLEU score of 37.3, a +1.0 improvement over the baseline. - Low-Resource Scenarios: The benefits were most pronounced when data was scarce. For English-Chinese (En-Zh) with only 10M sentences, the model saw a significant jump of +5.2 BLEU points (from 19.3 to 24.5).
- Latent Alignment: Analysis of word embedding similarity confirmed that these syntactic objectives lead to a much better-aligned bilingual space compared to standard unsupervised methods.
Discussion and Future Work
While leveraging Universal Grammar provides a powerful signal for UNMT, the method currently depends on high-quality constituent parsers, which may not be available for truly low-resource or endangered languages. Furthermore, the improvement from adding constituent trees eventually plateaus relative to the gains seen from true parallel data.
Future research will explore the use of dependency syntax, which is more widely available for over 100 languages via the Universal Dependencies project, potentially extending the reach of this structural bridging technique to a broader array of the world's languages.
Summary updated