Introduction
Neural Machine Translation (NMT) has traditionally relied on massive parallel corpora—identical texts in two languages—to achieve high performance. However, such "gold standard" data is scarce for most language pairs. While unsupervised NMT attempts to bridge this gap using monolingual data, it often requires complex pipelines like back-translation. This paper proposes a self-supervised approach that leverages comparable corpora, such as Wikipedia, where articles in different languages cover similar topics without being direct translations.
Methodology: Mining Parallelism from Comparable Data
The core innovation is a joint architecture that simultaneously selects training data and learns NMT representations. The process begins with word embeddings aligned using a minimal "seed" dictionary of numerals.
The system then enters an iterative selection loop:
- Representation: Sentences are represented using either word embedding sums ($C_e$) or encoder hidden state sums ($C_h$).
- Scoring: A margin-based function identifies unique, high-confidence translation pairs within linked Wikipedia articles, effectively penalizing "hubs" (sentences that appear similar to many others).
- Training: Selected pairs are immediately used for back-propagation, updating the model and refining the representations for the next iteration.
Crucially, this method avoids back-translation, relying instead on the model's evolving ability to "mine" existing parallel sentences from the comparable corpus.
The Doubly Virtuous Circle
The authors describe this process as a "doubly virtuous circle." As the NMT model improves, its internal representations become more semantically rich. This enhanced semantic understanding allows the model to identify more complex and accurate translation pairs in the next epoch. This new data, in turn, further boosts the model's translation capabilities.
This feedback loop also results in an emergent curriculum. Initially, the model selects simple sentences with shared anchors like dates or names. Over time, it graduates to identifying semantically equivalent sentences with no obvious surface-level similarities.
Key Results and Comparison
The self-supervised approach demonstrates significant gains over traditional unsupervised baselines. By using margin-based scoring and high-precision selection (requiring agreement between $C_e$ and $C_h$), the model achieves competitive BLEU scores on the newstest2014 benchmark.
The best-performing Transformer-based model reached 29.21 BLEU for English-to-French, outperforming established unsupervised NMT systems that typically range between 15-17 BLEU. The system successfully extracted approximately 2.4 million high-quality sentence pairs from Wikipedia.
Conclusion and Future Directions
While highly effective, the method currently depends on the existence of comparable corpora with semantic overlap. Future work may integrate back-translation to utilize "rejected" sentences that lack a direct match in the comparable corpus. Additionally, the authors see potential for this method in domain adaptation, where general models can be fine-tuned using specialized comparable datasets in fields like law or medicine.
Summary updated