TL;DR
Unsupervised cross-lingual word embedding (CLWE) methods usually fail when languages are typologically distant or trained on different domains because they falsely assume monolingual embedding spaces are structurally identical. This paper demonstrates that augmenting monolingual training data with pseudo-parallel text generated by an Unsupervised Machine Translation (UMT) system dramatically improves the structural similarity between spaces. This simple data-augmentation trick achieves state-of-the-art results on bilingual lexicon induction and various downstream cross-lingual tasks without requiring any actual parallel data.
What Problem Does This Paper Address?
To connect words across different languages without a dictionary, modern NLP relies on unsupervised CLWEs. Typically, researchers train word embeddings for Language A and Language B independently, then try to find a linear transformation (a rotation matrix) to align the two vector spaces.
However, this approach relies on a massive assumption: isomorphism—the idea that word embedding spaces across different languages share a nearly identical geometric structure. In reality, this assumption falls apart if the training corpora come from different domains or if the languages are structurally and typologically distant (like English and Japanese).
The authors tackle this fundamental bottleneck. Instead of trying to fix the alignment algorithm, they ask a more intuitive question: Can we alter the training text itself to force independently trained embedding spaces to look more alike?
Methods
The authors propose a straightforward pipeline that leverages Unsupervised Machine Translation (UMT) as a data augmentation tool:
- Train a UMT System: Using purely monolingual corpora, they train an unsupervised phrase-based statistical machine translation (USMT) model to translate between the source and target languages.
- Generate Pseudo Corpora: They use this UMT system to translate the source corpus into the target language, and vice-versa, creating "pseudo-parallel" datasets.
- Data Augmentation: They concatenate these automatically translated pseudo-sentences with the original monolingual training data.
- Independent Training & Mapping: Finally, they train standard monolingual embeddings (using
fastText) on this augmented text and align them using an off-the-shelf unsupervised mapping tool (VecMap).
Datasets
- Training Corpora: 10 million sentences randomly extracted from Wikipedia dumps for English (
en), French (fr), German (de), and Japanese (ja). - Bilingual Lexicon Induction (BLI) Evaluation:
XLing-Evalfor English-French and English-German; a Google Translate-generated dictionary for English-Japanese. - Downstream Cross-Lingual Evaluation:
MLDoc(Topic Classification)Webis-CLS-10(Sentiment Analysis)Universal Dependencies (UD) English EWT/PUD treebanks(Dependency Parsing)MultiNLI/XNLI(Natural Language Inference)- Monolingual Evaluation:
SimVerb-3500andMEN(Word Similarity).
Key Insights and Ideas
- The Power of Partial Parallelism: Why does adding machine-translated text help? First, it forces the content of the two corpora to overlap. If Language A's corpus talks about politics and Language B's talks about sports, UMT bridges the gap by copying those contexts into both languages.
- Traces of the Source Language: Even when translated, pseudo-sentences retain the underlying co-occurrence statistics and structural "fingerprints" of the original language. This acts as a geometric anchor, pulling the independently trained vector spaces into a shared structural alignment.
- More Data $\neq$ Better Data: Interestingly, augmenting both languages simultaneously isn't always optimal. The authors found that augmenting only one side often yielded the best results. This is because the two pseudo-corpora have different linguistic properties, and mixing them too aggressively can introduce conflicting signals.
Key Results
The proposed mapping method with pseudo-data augmentation consistently outperformed standard unsupervised mapping, joint training, and phrase-table induction methods.
- Bilingual Lexicon Induction (BLI): On English-to-French, the Precision@1 (P@1) score jumped from 0.612 (baseline mapping) to 0.684 (with pseudo-data). For the difficult English-to-Japanese pair, P@1 improved from 0.378 to 0.405.
- Downstream Tasks: In cross-lingual Topic Classification, the proposed method achieved the highest accuracy across all language pairs, showing statistically significant improvements particularly in
en-franden-ja. - Translation Quality Matters: The authors showed a direct correlation between the quality of the UMT system (measured by BLEU score) and the final alignment accuracy of the cross-lingual embeddings. Better translations yield better structural alignment.
Limitations and Future Work
While the method significantly improves cross-lingual alignment, it comes with a trade-off in monolingual quality. For typologically distant languages like Japanese, adding noisy pseudo-translations slightly degraded performance on monolingual word similarity tasks.
Additionally, the authors discovered a diminishing return when trying to feed these improved embeddings back into the UMT system for iterative improvement. Because the pseudo-corpora suffer from lower lexical diversity (a lower type-token ratio), the resulting embeddings become overly standardized, preventing the translation model from learning diverse sentence patterns during back-translation.
The authors suggest that exploring this data-augmentation technique within semi-supervised settings remains a promising direction for future research.
Summary updated