Data Augmentation with Unsupervised Machine Translation Improves the Structural Similarity of Cross-lingual Word Embeddings

Sosuke Nishikawa, Ryokan Ri, Yoshimasa Tsuruoka

TL;DR

This paper proposes a novel method to improve cross-lingual word embeddings (CLWEs) by augmenting training data with pseudo-parallel sentences generated by unsupervised machine translation (UMT). This augmentation enhances the structural similarity between monolingual embedding spaces, leading to better performance on bilingual lexicon induction and downstream cross-lingual tasks.

What Problem Does This Paper Address?

Cross-lingual word embeddings (CLWEs) aim to create a shared semantic space for words across different languages. Many unsupervised CLWE methods rely on mapping-based approaches, which assume that independently trained monolingual embedding spaces have similar structures. However, this assumption often doesn't hold true, especially when corpora differ in domain or language typology, leading to suboptimal alignment. Existing work has focused on improving structural similarity before mapping, but this paper explores how to leverage text data itself for this purpose.

Methods

The core of this paper's contribution lies in using Unsupervised Machine Translation (UMT) for data augmentation. Here's the breakdown:

  1. UMT Model Training: They first train UMT models for each language pair using monolingual corpora. This involves a word-by-word translation model, a language model, and a back-translation mechanism to iteratively refine the translation quality.
  2. Pseudo-Parallel Corpus Generation: The trained UMT models are then used to translate the original monolingual training corpora, creating "pseudo-parallel" corpora.
  3. Monolingual Embedding Training: These pseudo-parallel corpora are concatenated with the original corpora. Monolingual word embeddings are then trained independently on these augmented datasets.
  4. Mapping-Based CLWE: Finally, these enhanced monolingual embeddings are aligned into a shared CLWE space using a standard unsupervised mapping algorithm (specifically, the one proposed by Artetxe et al., 2018a).

The key novelty is using UMT-generated data not for joint training but for augmenting the monolingual training data before the mapping step.

Datasets

The authors used Wikipedia dumps for the following language pairs:

  • English-French (en-fr)
  • English-German (en-de)
  • English-Japanese (en-ja)

They extracted 10 million sentences for each language. For evaluation, they used:

  • Cross-Domain Similarity Local Scaling (Lample et al., 2018b) for identifying translation pairs in embedding spaces.
  • XLing-Eval test sets for En-Fr and En-Ge.
  • Google Translate for automatically creating word dictionaries for En-Ja.
  • MLDoc corpus for topic classification.
  • Webis-CLS-10 corpus for sentiment analysis.
  • UD English EWT dataset and PUD treebanks for dependency parsing.
  • English MultiNLI corpus for training and multilingual XNLI corpus for evaluation of natural language inference.

Key Insights and Ideas

  • UMT as a Bridge: The paper demonstrates that UMT-generated pseudo-parallel data can effectively bridge the structural gap between monolingual embedding spaces, which is a major hurdle for mapping-based CLWE methods.
  • Structural Similarity is Key: The core hypothesis is that augmenting corpora with UMT output improves the structural similarity of monolingual embeddings, making them more amenable to linear alignment.
  • Content Sharing Matters: The pseudo-parallel data helps by making the source and target corpora partially parallel, allowing for better alignment.
  • Reflecting Original Language Statistics: The translated sentences retain traces of the original language's co-occurrence statistics, which aids in learning more aligned embedding structures.
  • Augmentation Strategy is Nuanced: Surprisingly, augmenting only the source or target, rather than both, sometimes yielded better results, suggesting that the specific nature and interaction of the pseudo-corpora are important.

Key Results

  • Improved Bilingual Lexicon Induction (BLI): The proposed method consistently outperformed baseline approaches on BLI tasks across all language pairs, with significant improvements observed when augmenting both source and target corpora.
  • Downstream Task Performance: The CLWEs trained with the UMT-augmented data showed improved performance on topic classification, sentiment analysis, dependency parsing, and natural language inference tasks, especially for topic classification.
  • UMT Quality Matters: The quality of the UMT model directly impacts CLWE performance; better UMT translations lead to better CLWEs.
  • Monolingual Embedding Quality: While the pseudo-corpus slightly degraded monolingual word similarity scores in some cases (particularly for Japanese), the overall CLWE performance still improved, indicating that the structure optimized for cross-lingual mapping is more critical than perfect monolingual similarity.

Limitations and Future Work

  • UMT Quality Dependency: The method's effectiveness is tied to the quality of the UMT system. Lower-quality UMT can lead to less effective augmentation.
  • Monolingual Degradation: In some language pairs (e.g., Japanese), monolingual embedding quality slightly decreased, suggesting potential trade-offs.
  • Future Work: The authors suggest exploring the use of their method in semi-supervised settings, as the improvements in structural similarity could be beneficial when some cross-lingual supervision is available. They also note that while their method improves performance, the exact mechanisms and optimal augmentation strategies warrant further investigation.

Summary updated

Abstract

Unsupervised cross-lingual word embedding (CLWE) methods learn a linear transformation matrix that maps two monolingual embedding spaces that are separately trained with monolingual corpora. This method relies on the assumption that the two embedding spaces are structurally similar, which does not necessarily hold true in general. In this paper, we argue that using a pseudo-parallel corpus generated by an unsupervised machine translation model facilitates the structural similarity of the two embedding spaces and improves the quality of CLWEs in the unsupervised mapping method. We show that our approach outperforms other alternative approaches given the same amount of data, and, through detailed analysis, we show that data augmentation with the pseudo data from unsupervised machine translation is especially effective for mapping-based CLWEs because (1) the pseudo data makes the source and target corpora (partially) parallel; (2) the pseudo data contains information on the original language that helps to learn similar embedding spaces between the source and target languages.
BibTeX
@inproceedings{nishikawa2021data,
  author = {Sosuke Nishikawa and Ryokan Ri and Yoshimasa Tsuruoka},
  title = {{Data Augmentation with Unsupervised Machine Translation Improves the Structural Similarity of Cross-lingual Word Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure