Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining

Ivana Kvapilíková, Mikel Artetxe, Gorka Labaka, Eneko Agirre, Ondrej Bojar

TL;DR

This paper introduces a fully unsupervised approach to derive multilingual sentence embeddings without requiring human-translated parallel text. By generating a small synthetic parallel corpus using unsupervised machine translation (MT) and fine-tuning a pretrained XLM model via a Translation Language Model (TLM) objective, the model achieves dramatic cross-lingual alignment gains—even boosting performance on completely unrelated language pairs.

What Problem Does This Paper Address?

Mining parallel sentences from non-parallel, comparable corpora is crucial for training machine translation systems and other cross-lingual NLP tools, particularly for low-resource languages. State-of-the-art approaches like LASER rely on massive parallel corpora (hundreds of millions of sentence pairs) to align embeddings. Meanwhile, unsupervised pretrained language models like multilingual BERT or vanilla XLM develop shared cross-lingual spaces, but their internal representations remain imperfectly aligned and language-dependent. The authors tackle the challenge of bridging this alignment gap using strictly monolingual data.

Methods

The proposed framework combines unsupervised machine translation with cross-lingual language model fine-tuning:

  • Synthetic Bitext Generation: The authors train an unsupervised Transformer MT model (using denoising autoencoding and back-translation) on monolingual news text. This system translates 10k sentences per language to produce a tiny synthetic bilingual corpus (20k sentence pairs).
  • TLM Fine-Tuning: A publicly available pretrained XLM-100 model is fine-tuned for just a single epoch using the Translation Language Model (TLM) objective on the synthetic pairs. In TLM, paired sentences are concatenated with masked tokens, forcing self-attention to leverage context across languages.
  • Sentence Embedding Extraction: Embeddings are derived via element-wise mean-pooling of subword representations from the 12th layer (identified as the optimal layer for cross-lingual tasks).
  • Scoring Candidates: Sentence pairs are retrieved and ranked using a margin-based cosine similarity measure to overcome hubness in high-dimensional embedding spaces.

Datasets

  • Pretraining & MT Training: XLM-100 pretrained on Wikipedia (100 languages). Unsupervised MT systems trained on NewsCrawl 2007–2008 (5M sentences/language). Fine-tuning data sampled from NewsCrawl 2018 (10k sentences each for Cs, De, En).
  • BUCC 2018 Shared Task: The standard comparable corpus mining benchmark containing parallel sentences injected into Wikipedia text across four language pairs (en-de, en-fr, en-ru, en-zh).
  • News Mining Benchmark (Introduced): Created by shuffling 10k parallel sentences from News Commentary into 400k monolingual sentences from NewsCrawl. This eliminates the evaluation bias of XLM having seen BUCC's Wikipedia sentences during pretraining.
  • Parallel Sentence Matching (PSM): Multi-way evaluation using newstest2012 / newstest2013 (6 languages) and the Tatoeba test set (over 100 languages aligned with English). Additional small authentic sets were used for analysis: Flores (en-ne, 5k pairs) and News Commentary (en-kk, 10k pairs).

Key Insights and Ideas

  • Cross-Lingual Alignment Propagation: Fine-tuning on a single synthetic language pair (e.g., English–German) significantly improves sentence representations across completely unrelated language pairs (e.g., Russian–Spanish, Czech–Chinese). Aligning one pair via shared Transformer weights pulls the broader multilingual space together.
  • Synthetic Data Matches Authentic Data: Using just 20k pairs of noisy, synthetic translations from unsupervised MT yielded results comparable to fine-tuning on genuine, human-translated bitext.
  • Layer Depth Matters: Contextual representation alignment peaks around internal layer 12 (5th-to-last layer). Fine-tuning predominantly alters the deepest layers while leaving early semantic representations stable.
  • Tiny Data is Sufficient: Massive parallel corpora are unnecessary; development accuracy actually deteriorated when fine-tuning on datasets larger than 20k pairs.

Key Results

  • BUCC 2018 Mining: The proposed method gained 14.0 to 22.3 F1 points over vanilla XLM across all evaluated pairs, scoring 80.06 (en-de), 78.77 (en-fr), 77.16 (en-ru), and 67.04 (en-zh). It crushed the unsupervised word-mapping baseline (VecMap; 17.68–32.94 F1) and outperformed several supervised systems.
  • News Test Mining: Achieved +6.2 to +11.4 F1 points over vanilla XLM, demonstrating that the performance gain is robust and not merely an artifact of pretraining domain overlap.
  • Sentence Matching Accuracy:
  • On newstest, the method achieved 90%–95% accuracy across 8 language pairs (~7% improvement over vanilla XLM).
  • On Tatoeba, the approach yielded average improvements of roughly 20 percentage points over vanilla XLM, rivaling the supervised LASER baseline on low-resource languages such as Kazakh, Georgian, and Nepali.

Limitations and Future Work

  • Performance Gap with Large Supervised Models: While beating older supervised pipelines, the approach still lags behind massive supervised dual-encoders like LASER on high-resource language pairs (e.g., 80.06 vs. 96.19 F1 on BUCC en-de).
  • Pretraining Sensitivity: Retrieval accuracy drops sharply for languages with minimal representation in the underlying XLM pretraining corpus (e.g., Wikipedia).
  • Future Directions: The authors plan to test whether this unsupervised alignment propagation transfers benefits to downstream cross-lingual tasks such as Cross-lingual Natural Language Inference (XNLI) and cross-lingual Semantic Textual Similarity (STS).

Summary updated

Abstract

Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. We propose a novel unsupervised method to derive multilingual sentence embeddings relying only on monolingual data. We first produce a synthetic parallel corpus using unsupervised machine translation, and use it to fine-tune a pretrained cross-lingual masked language model (XLM) to derive the multilingual sentence representations. The quality of the representations is evaluated on two parallel corpus mining tasks with improvements of up to 22 F1 points over vanilla XLM. In addition, we observe that a single synthetic bilingual corpus is able to improve results for other language pairs.
BibTeX
@inproceedings{kvapilikova2020unsupervised,
  author = {Ivana Kvapilíková and Mikel Artetxe and Gorka Labaka and Eneko Agirre and Ondrej Bojar},
  title = {{Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=80DAUVkBV-S},
}

Loading related content...

Figure