Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings

Mikel Artetxe, Holger Schwenk

Authors: Mikel Artetxe, Holger Schwenk. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).

Introduction

Neural Machine Translation (NMT) systems are notoriously data-hungry, requiring millions of parallel sentence pairs to achieve high performance. While the internet offers a vast repository of multilingual text, most of it is "comparable" (discussing similar topics) rather than strictly "parallel" (direct translations). Parallel corpus mining—the process of extracting translation pairs from these massive, noisy datasets—is therefore a critical challenge in modern NLP.

Traditional mining methods often rely on absolute cosine similarity between sentence embeddings. However, this approach suffers from "scale inconsistency" and the "hubness" problem, where certain vectors appear close to many others in high-dimensional space, leading to high false-positive rates. This paper introduces a relative, margin-based scoring approach that significantly improves the accuracy of parallel sentence retrieval.

Methodology

The authors' approach consists of two primary components: a robust multilingual embedding space and a novel scoring function.

Multilingual Sentence Embeddings

The system utilizes a language-agnostic BiLSTM encoder-decoder architecture, which later became the foundation for the LASER (Language-Agnostic SEntence Representations) toolkit. The encoder maps sentences from any supported language into a shared vector space, ensuring that sentences with similar semantic meanings are positioned closely together regardless of the source language.

The BiLSTM encoder-decoder architecture used to generate language-agnostic sentence embeddings.

Margin-based Scoring

The core innovation is the transition from absolute similarity to a margin-based metric. Instead of evaluating the raw cosine similarity between a source sentence $x$ and a target sentence $y$, the system calculates a score relative to the average similarity of their $k$-nearest neighbors.

Three variants were tested: Absolute (baseline cosine similarity), Distance (the difference between the candidate score and the neighborhood average), and Ratio. The Ratio method proved most effective, as it naturally penalizes "hubs"—sentences that are vaguely similar to many others—by normalizing the score based on the local density of the embedding space.

Experimental Results

The proposed method was evaluated on several benchmarks, including the BUCC mining task and UN corpus reconstruction. The results demonstrated a transformative improvement over previous state-of-the-art techniques.

Performance comparison of Absolute, Distance, and Ratio scoring functions across different retrieval directions.

As shown in the table above, the Ratio scoring function consistently outperformed the absolute baseline across multiple metrics and language pairs. On the BUCC task, the method achieved a massive jump of over 10 F1 points. Furthermore, when used to filter the noisy ParaCrawl dataset for English-German NMT training, the method yielded a 1.1 BLEU point improvement over the best existing filtered versions, demonstrating that higher-quality mining directly translates to better downstream model performance.

Key Insights and Limitations

  • Relative Context Matters: The "aha moment" of the research is that the context of a match is more important than the raw score. A high-confidence match is one where the similarity is significantly higher than any other potential neighbor.
  • Simplicity and Scalability: Unlike previous solutions that required training complex secondary classifiers to rescale scores, the margin-based formula is mathematically simple and requires no additional supervised training.
  • The Paraphrase Penalty: One noted limitation is that the margin approach can inadvertently penalize sentences with many valid paraphrases in the corpus. If a sentence has multiple correct translations present, the "margin" between the best match and the other valid neighbors will be small, potentially leading the system to filter out legitimate pairs.

Conclusion

By addressing the scale inconsistency of absolute similarity scores, this margin-based approach provides a robust and scalable solution for parallel corpus mining. The method has since been integrated into large-scale data pipelines, enabling the creation of high-quality translation datasets for dozens of languages.

Summary updated

BibTeX
@inproceedings{artetxe2019margin,
  author = {Mikel Artetxe and Holger Schwenk},
  title = {{Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=zh1Pw3QCzHK},
}
Figure