Abstract
Machine translation is highly sensitive to the size and quality of the training data, which has led to an increasing interest in collecting and filtering large parallel corpora. In this paper, we propose a new method for this task based on multilingual sentence embeddings. In contrast to previous approaches, which rely on nearest neighbor retrieval with a hard threshold over cosine similarity, our proposed method accounts for the scale inconsistencies of this measure, considering the margin between a given sentence pair and its closest candidates instead. Our experiments show large improvements over existing methods. We outperform the best published results on the BUCC mining task and the UN reconstruction task by more than 10 F1 and 30 precision points, respectively. Filtering the English-German ParaCrawl corpus with our approach, we obtain 31.2 BLEU points on newstest2014, an improvement of more than one point over the best official filtered version.
Introduction
Neural Machine Translation (NMT) is notoriously data-hungry. To achieve high-quality translation, models require millions of parallel sentence pairs. While the internet provides a vast repository of text, most available data is "comparable" (sharing topics) rather than strictly "parallel" (direct translations).
The standard approach to identifying parallel sentences involves using multilingual sentence embeddings to map text into a shared vector space, then using cosine similarity to find close pairs. However, this method suffers from a "hubness" problem: some sentences act as hubs, exhibiting high similarity scores with many unrelated candidates. This inconsistency makes it impossible to set a global similarity threshold that effectively filters noise without discarding valid translations.
Methodology
To address the limitations of absolute similarity, this research introduces a relative margin-based scoring function. The core idea is to measure how much better a translation candidate is compared to the "next best" neighbors in the embedding space.
Embedding Architecture
The system utilizes a BiLSTM encoder-decoder architecture to generate 1024-dimensional sentence embeddings. The encoder is designed to be language-agnostic, mapping sentences from diverse languages into a unified representation space.
The Margin Function
Instead of relying solely on the cosine similarity between sentence $x$ and sentence $y$, the proposed method evaluates that similarity relative to the average similarity of $x$ and $y$ with their respective $k$-nearest neighbors in the target language. The researchers tested several variants, finding that a Ratio margin function—calculating the ratio between the pair's similarity and the average neighbor similarity—consistently outperformed simple distance-based subtractions. This ratio effectively normalizes scores across the entire corpus, mitigating the hubness effect.
Experimental Setup
The method was evaluated across three distinct tasks:
- BUCC Mining Task: A benchmark for bitext mining across English, German, French, Russian, and Chinese.
- UN Corpus Reconstruction: A challenge involving the realignment of 11.3 million shuffled sentences from United Nations documents.
- ParaCrawl Filtering: Applying the method to filter the notoriously noisy English-German ParaCrawl web-crawled dataset to measure its impact on downstream NMT performance.
Results and Discussion
The margin-based approach yielded substantial improvements over previous state-of-the-art techniques. On the BUCC benchmark, the method achieved a massive jump in performance, outperforming previous records by over 10 F1 points in several categories.
In the UN Reconstruction task, the system achieved 83-85% precision in identifying exact original sentence pairs, nearly doubling the performance of previous high-performing systems (which reached ~48-54%). Furthermore, filtering the ParaCrawl dataset resulted in an NMT model scoring 31.2 BLEU points on newstest2014, a 1.1-point improvement over the best official filtered version.
Key Insights
- Scale Over Value: The success of the margin function demonstrates that the relative scale of similarity is more informative than the absolute value.
- Bidirectionality: Considering neighbors in both source-to-target and target-to-source directions provides robustness against false positives.
- Geometric Simplicity: Unlike methods requiring complex classifiers, this approach relies on the inherent geometry of the embedding space, making it highly scalable.
Conclusion and Future Work
While highly effective, the authors note that the margin-based approach may occasionally penalize sentences with many valid paraphrases in a corpus, as the "margin" between the best and fifth-best match might be small. However, this is statistically rare in standard datasets. The release of the LASER (Language-Agnostic SEntence Representations) toolkit, which supports over 90 languages, provides a powerful resource for mining parallel data in low-resource languages where human-translated corpora are currently scarce.
Summary updated