Self-Supervised Neural Machine Translation

Dana Ruiter, Cristina España-Bonet, Josef van Genabith

Introduction

Neural Machine Translation (NMT) has traditionally relied on massive parallel corpora—identical texts in two languages—to achieve high performance. However, such "gold standard" data is scarce for most language pairs. While unsupervised NMT attempts to bridge this gap using monolingual data, it often requires complex pipelines like back-translation. This paper proposes a self-supervised approach that leverages comparable corpora, such as Wikipedia, where articles in different languages cover similar topics without being direct translations.

Methodology: Mining Parallelism from Comparable Data

The core innovation is a joint architecture that simultaneously selects training data and learns NMT representations. The process begins with word embeddings aligned using a minimal "seed" dictionary of numerals.

The system then enters an iterative selection loop:

  1. Representation: Sentences are represented using either word embedding sums ($C_e$) or encoder hidden state sums ($C_h$).
  2. Scoring: A margin-based function identifies unique, high-confidence translation pairs within linked Wikipedia articles, effectively penalizing "hubs" (sentences that appear similar to many others).
  3. Training: Selected pairs are immediately used for back-propagation, updating the model and refining the representations for the next iteration.

Crucially, this method avoids back-translation, relying instead on the model's evolving ability to "mine" existing parallel sentences from the comparable corpus.

The Doubly Virtuous Circle

The authors describe this process as a "doubly virtuous circle." As the NMT model improves, its internal representations become more semantically rich. This enhanced semantic understanding allows the model to identify more complex and accurate translation pairs in the next epoch. This new data, in turn, further boosts the model's translation capabilities.

BLEU score improvement over training epochs for English-to-French (en2fr) and French-to-English (fr2en) tasks.

This feedback loop also results in an emergent curriculum. Initially, the model selects simple sentences with shared anchors like dates or names. Over time, it graduates to identifying semantically equivalent sentences with no obvious surface-level similarities.

Key Results and Comparison

The self-supervised approach demonstrates significant gains over traditional unsupervised baselines. By using margin-based scoring and high-precision selection (requiring agreement between $C_e$ and $C_h$), the model achieves competitive BLEU scores on the newstest2014 benchmark.

Comparison of BLEU scores across different NMT methods. The self-supervised Transformer models significantly outperform unsupervised baselines.

The best-performing Transformer-based model reached 29.21 BLEU for English-to-French, outperforming established unsupervised NMT systems that typically range between 15-17 BLEU. The system successfully extracted approximately 2.4 million high-quality sentence pairs from Wikipedia.

Conclusion and Future Directions

While highly effective, the method currently depends on the existence of comparable corpora with semantic overlap. Future work may integrate back-translation to utilize "rejected" sentences that lack a direct match in the comparable corpus. Additionally, the authors see potential for this method in domain adaptation, where general models can be fine-tuned using specialized comparable datasets in fields like law or medicine.

Summary updated

BibTeX
@inproceedings{ruiter2019self,
  author = {Dana Ruiter and Cristina España-Bonet and Josef van Genabith},
  title = {{Self-Supervised Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1178},
}
Figure