Cross-lingual Retrieval for Iterative Self-Supervised Training

Chau Tran, Yuqing Tang, Xian Li, Jiatao Gu

Abstract

CRISS (Cross-lingual Retrieval for Iterative Self-Supervised training) is a novel framework that enables AI models to learn translation between languages using only unlabeled, monolingual text. By iteratively mining its own parallel sentence pairs and retraining, the model achieves state-of-the-art results in unsupervised translation and cross-lingual retrieval. Notably, CRISS often outperforms supervised methods in low-resource language scenarios, providing a scalable solution for the thousands of languages lacking massive human-translated datasets.

Introduction: The Challenge of Low-Resource Translation

Traditional Machine Translation (MT) relies on massive parallel corpora—millions of human-translated sentence pairs. However, such datasets are unavailable for the vast majority of the world's 7,000+ languages. While unsupervised MT exists, current techniques like back-translation are computationally intensive and often produce noisy training data. CRISS addresses this by exploiting "emergent alignment": the phenomenon where multilingual models like mBART naturally align internal representations across languages, even when trained on separate monolingual data. The authors set out to leverage this alignment to create a more efficient, iterative bootstrapping process.

Methodology: The CRISS Framework

The CRISS framework operates as an iterative loop designed to refine translation capabilities from monolingual data.

The CRISS iterative framework for unsupervised parallel data mining and multilingual training.

The process consists of two primary phases:

  1. Unsupervised Parallel Data Mining: The model takes massive amounts of monolingual data and uses its encoder to convert sentences into semantic vectors. Using a margin-based scoring function and K-Nearest Neighbors (KNN) search, it identifies "pseudo-parallel" sentence pairs across different languages.
  2. Multilingual Training: The model is then trained as a standard sequence-to-sequence (seq2seq) transformer on these mined pairs. Crucially, it uses a language-agnostic objective, learning to generate target sentences regardless of the input language.

This process is iterative: as the model trains, its embeddings become better aligned, which in turn allows for the mining of higher-quality parallel data in the subsequent cycle.

Experimental Setup

The model was trained on monolingual data for 25 languages subsampled from Common Crawl (up to 100M sentences per language). Evaluation spanned several key benchmarks:

  • Retrieval: The Tatoeba dataset (part of the XTREME benchmark) and TED58.
  • Translation: Standard benchmarks including WMT (English-German, English-French, English-Romanian), Flores (English-Nepali, English-Sinhala), and IWSLT (English-Japanese).

Key Insights: Emergent Alignment and Self-Correction

A significant finding is the "Universal Boost": finetuning the model on just one pair of languages improves cross-lingual alignment for all other languages in the model's vocabulary. Furthermore, the authors found that mining pairs relative to a few "pivot" languages (such as English, Hindi, Spanish, or Chinese) is sufficient to align the entire embedding space. This self-correcting cycle allows the model to start with a strong multilingual foundation (mBART) and progressively refine its translation accuracy through successive iterations.

Results: Setting New Benchmarks

CRISS achieved state-of-the-art results in 9 out of 10 unsupervised MT directions tested, showing an average improvement of 2.4 BLEU points over previous methods.

Unsupervised machine translation performance (BLEU) across three iterations of CRISS compared to baselines.

In sentence retrieval, CRISS improved absolute accuracy on the Tatoeba benchmark by a staggering 21.5% on average across 16 languages. In some low-resource scenarios, such as Kazakh, the unsupervised CRISS model actually outperformed existing supervised models (77.9% vs 17.39% accuracy in retrieval). Additionally, even when human-labeled data is available, using CRISS as a pre-training step provides an additional 1.8 BLEU boost over standard mBART.

Limitations and Future Directions

A primary limitation identified by the authors is the risk of "near-miss" matches. Because the model mines based on semantic similarity, it may pair sentences that are contextually similar but factually different (e.g., "I want to live in the US" vs. "I want to live in Canada"). Overfitting on these near-misses can lead to entity hallucinations. Future work will explore how these language-agnostic representations emerge during pre-training and extend the CRISS approach to non-seq2seq tasks, such as unsupervised structural discovery in linguistics.

Summary updated

BibTeX
@inproceedings{tran2020cross,
  author = {Chau Tran and Yuqing Tang and Xian Li and Jiatao Gu},
  title = {{Cross-lingual Retrieval for Iterative Self-Supervised Training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=rhrNqB8g0oS},
}
Figure