Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining

Ivana Kvapilíková, Mikel Artetxe, Gorka Labaka, Eneko Agirre, Ondrej Bojar

Abstract

Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. We propose a novel unsupervised method to derive multilingual sentence embeddings relying only on monolingual data. We first produce a synthetic parallel corpus using unsupervised machine translation, and use it to fine-tune a pretrained cross-lingual masked language model (XLM) to derive the multilingual sentence representations. The quality of the representations is evaluated on two parallel corpus mining tasks with improvements of up to 22 F1 points over vanilla XLM. In addition, we observe that a single synthetic bilingual corpus is able to improve results for other language pairs.

Introduction

Multilingual sentence embeddings are essential for cross-lingual tasks like translation and search. These embeddings map sentences from different languages into a shared mathematical space where semantically similar sentences cluster together. However, state-of-the-art models typically require millions of professionally translated sentence pairs, which are unavailable for many low-resource languages. While pretrained models like XLM are trained on vast monolingual data, their internal representations of different languages are often poorly aligned. This paper introduces a fully unsupervised pipeline to align these language spaces without human-translated text.

Methodology

The proposed method forces language alignment through a multi-step process:

  1. Unsupervised Machine Translation (MT): A Transformer-based MT system is trained using only monolingual data (e.g., NewsCrawl) via techniques like back-translation.
  2. Synthetic Corpus Generation: This MT system generates a "synthetic" parallel corpus of approximately 20,000 sentence pairs. Despite being noisy, this data provides sufficient signal for alignment.
  3. TLM Fine-tuning: A pretrained XLM-100 model is fine-tuned using the Translation Language Model (TLM) objective. By masking words in concatenated synthetic pairs, the model learns to use cross-lingual context to fill blanks, effectively "stitching" the language spaces together.
  4. Layer Selection: Analysis of the model's internal layers revealed that the 12th layer (out of 16) provides the most effective semantic representations for sentence matching.
Average accuracy across XLM layers for the proposed method vs. vanilla XLM.

Key Results and Insights

The method was evaluated on the BUCC 2018 shared task (parallel sentence mining) and the Tatoeba dataset (sentence matching).

Performance Gains

On the BUCC mining task, the proposed method achieved an F1 score of 80.06 for English-German, a significant improvement over the 62.10 baseline. Across various language pairs, the method improved F1 scores by up to 22 points.

Performance comparison on the BUCC 2018 shared task for parallel corpus mining.

The Propagation Effect

A key finding is the "propagation effect": fine-tuning on a single synthetic language pair (e.g., English-German) improves alignment for all other languages in the model's shared parameter space. This suggests that aligning one part of the model "pulls" other languages into better alignment.

Quality vs. Quantity

The results demonstrate that massive parallel datasets are not always necessary. Just 20,000 synthetic pairs were enough to significantly boost performance, outperforming older unsupervised methods based on static word embeddings.

Conclusion and Future Work

While this method significantly narrows the gap between unsupervised and supervised models, it still relies on sufficient monolingual data for initial pretraining. Future work will explore applying this alignment technique to downstream reasoning tasks like Cross-lingual Natural Language Inference (XNLI).

Summary updated

Abstract

Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. We propose a novel unsupervised method to derive multilingual sentence embeddings relying only on monolingual data. We first produce a synthetic parallel corpus using unsupervised machine translation, and use it to fine-tune a pretrained cross-lingual masked language model (XLM) to derive the multilingual sentence representations. The quality of the representations is evaluated on two parallel corpus mining tasks with improvements of up to 22 F1 points over vanilla XLM. In addition, we observe that a single synthetic bilingual corpus is able to improve results for other language pairs.
BibTeX
@inproceedings{kvapilikova2020unsupervised,
  author = {Ivana Kvapilíková and Mikel Artetxe and Gorka Labaka and Eneko Agirre and Ondrej Bojar},
  title = {{Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=80DAUVkBV-S},
}
Figure