Data Augmentation with Unsupervised Machine Translation Improves the Structural Similarity of Cross-lingual Word Embeddings

Sosuke Nishikawa, Ryokan Ri, Yoshimasa Tsuruoka

TL;DR

Unsupervised cross-lingual word embedding (CLWE) methods usually fail when languages are typologically distant or trained on different domains because they falsely assume monolingual embedding spaces are structurally identical. This paper demonstrates that augmenting monolingual training data with pseudo-parallel text generated by an Unsupervised Machine Translation (UMT) system dramatically improves the structural similarity between spaces. This simple data-augmentation trick achieves state-of-the-art results on bilingual lexicon induction and various downstream cross-lingual tasks without requiring any actual parallel data.

What Problem Does This Paper Address?

To connect words across different languages without a dictionary, modern NLP relies on unsupervised CLWEs. Typically, researchers train word embeddings for Language A and Language B independently, then try to find a linear transformation (a rotation matrix) to align the two vector spaces.

However, this approach relies on a massive assumption: isomorphism—the idea that word embedding spaces across different languages share a nearly identical geometric structure. In reality, this assumption falls apart if the training corpora come from different domains or if the languages are structurally and typologically distant (like English and Japanese).

The authors tackle this fundamental bottleneck. Instead of trying to fix the alignment algorithm, they ask a more intuitive question: Can we alter the training text itself to force independently trained embedding spaces to look more alike?

Methods

The authors propose a straightforward pipeline that leverages Unsupervised Machine Translation (UMT) as a data augmentation tool:

  1. Train a UMT System: Using purely monolingual corpora, they train an unsupervised phrase-based statistical machine translation (USMT) model to translate between the source and target languages.
  2. Generate Pseudo Corpora: They use this UMT system to translate the source corpus into the target language, and vice-versa, creating "pseudo-parallel" datasets.
  3. Data Augmentation: They concatenate these automatically translated pseudo-sentences with the original monolingual training data.
  4. Independent Training & Mapping: Finally, they train standard monolingual embeddings (using fastText) on this augmented text and align them using an off-the-shelf unsupervised mapping tool (VecMap).
Figure 1: Our framework for training CLWEs using un- supervised machine translation (UMT). We first train UMT models using monolingual corpora for each lan- guage. We then translate all the training corpora and concatenate the outputs with the original corpora, and train monolingual word embeddings independently. Fi- nally, we map these word embeddings on a shared em- bedding.

Datasets

  • Training Corpora: 10 million sentences randomly extracted from Wikipedia dumps for English (en), French (fr), German (de), and Japanese (ja).
  • Bilingual Lexicon Induction (BLI) Evaluation: XLing-Eval for English-French and English-German; a Google Translate-generated dictionary for English-Japanese.
  • Downstream Cross-Lingual Evaluation:
  • MLDoc (Topic Classification)
  • Webis-CLS-10 (Sentiment Analysis)
  • Universal Dependencies (UD) English EWT / PUD treebanks (Dependency Parsing)
  • MultiNLI / XNLI (Natural Language Inference)
  • Monolingual Evaluation: SimVerb-3500 and MEN (Word Similarity).

Key Insights and Ideas

  • The Power of Partial Parallelism: Why does adding machine-translated text help? First, it forces the content of the two corpora to overlap. If Language A's corpus talks about politics and Language B's talks about sports, UMT bridges the gap by copying those contexts into both languages.
  • Traces of the Source Language: Even when translated, pseudo-sentences retain the underlying co-occurrence statistics and structural "fingerprints" of the original language. This acts as a geometric anchor, pulling the independently trained vector spaces into a shared structural alignment.
  • More Data $\neq$ Better Data: Interestingly, augmenting both languages simultaneously isn't always optimal. The authors found that augmenting only one side often yielded the best results. This is because the two pseudo-corpora have different linguistic properties, and mixing them too aggressively can introduce conflicting signals.

Key Results

The proposed mapping method with pseudo-data augmentation consistently outperformed standard unsupervised mapping, joint training, and phrase-table induction methods.

  • Bilingual Lexicon Induction (BLI): On English-to-French, the Precision@1 (P@1) score jumped from 0.612 (baseline mapping) to 0.684 (with pseudo-data). For the difficult English-to-Japanese pair, P@1 improved from 0.378 to 0.405.
  • Downstream Tasks: In cross-lingual Topic Classification, the proposed method achieved the highest accuracy across all language pairs, showing statistically significant improvements particularly in en-fr and en-ja.
  • Translation Quality Matters: The authors showed a direct correlation between the quality of the UMT system (measured by BLEU score) and the final alignment accuracy of the cross-lingual embeddings. Better translations yield better structural alignment.

Limitations and Future Work

While the method significantly improves cross-lingual alignment, it comes with a trade-off in monolingual quality. For typologically distant languages like Japanese, adding noisy pseudo-translations slightly degraded performance on monolingual word similarity tasks.

Additionally, the authors discovered a diminishing return when trying to feed these improved embeddings back into the UMT system for iterative improvement. Because the pseudo-corpora suffer from lower lexical diversity (a lower type-token ratio), the resulting embeddings become overly standardized, preventing the translation model from learning diverse sentence patterns during back-translation.

The authors suggest that exploring this data-augmentation technique within semi-supervised settings remains a promising direction for future research.

Summary updated

Abstract

Unsupervised cross-lingual word embedding (CLWE) methods learn a linear transformation matrix that maps two monolingual embedding spaces that are separately trained with monolingual corpora. This method relies on the assumption that the two embedding spaces are structurally similar, which does not necessarily hold true in general. In this paper, we argue that using a pseudo-parallel corpus generated by an unsupervised machine translation model facilitates the structural similarity of the two embedding spaces and improves the quality of CLWEs in the unsupervised mapping method. We show that our approach outperforms other alternative approaches given the same amount of data, and, through detailed analysis, we show that data augmentation with the pseudo data from unsupervised machine translation is especially effective for mapping-based CLWEs because (1) the pseudo data makes the source and target corpora (partially) parallel; (2) the pseudo data contains information on the original language that helps to learn similar embedding spaces between the source and target languages.
BibTeX
@inproceedings{nishikawa2021data,
  author = {Sosuke Nishikawa and Ryokan Ri and Yoshimasa Tsuruoka},
  title = {{Data Augmentation with Unsupervised Machine Translation Improves the Structural Similarity of Cross-lingual Word Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure