TL;DR
This paper introduces a novel unsupervised method for identifying parallel sentences in large, noisy web-scraped corpora, crucial for training machine translation systems. Their Mahalanobis distance-based approach effectively measures sentence similarity without relying on pre-existing clean parallel data, achieving competitive results in the WMT18 shared task, especially for lower-resource scenarios.
What Problem Does This Paper Address?
Training effective machine translation (MT) systems requires large amounts of parallel text – sentences that are translations of each other. While clean parallel corpora exist for high-resource languages, web-scraped data, though abundant, is often noisy, containing non-parallel sentences, duplicate content, and even non-linguistic data. Filtering this noisy data to extract high-quality parallel sentence pairs is a significant challenge. Traditional approaches often rely on supervised learning, requiring existing "clean" parallel data to train a model. However, for many language pairs, such clean data is scarce or non-existent, making unsupervised methods essential. This paper tackles the problem of identifying parallel sentences in a large, low-precision corpus without access to additional clean parallel data, a common scenario for lower-resource languages.
Methods
The core of the NRC submission is an unsupervised method for measuring sentence parallelism based on Mahalanobis distances. The intuition is that if two sentences are translations of each other, their vector representations should be "normal" or expected within the distribution of sentence pairs. Conversely, pairs that are "strange" or improbable are likely not parallel.
Here's a breakdown of their approach:
- Sentence Embeddings: They first train monolingual sentence embeddings using
sent2vecon all available monolingual data, including the "clean" parallel data (treated as two separate monolingual corpora). They experimented with various dimensions and settled on 300-dimensional vectors. - Pre-filtering: Before applying their core parallelism measure, they perform several pre-filtering steps to clean the data further. This includes:
- Additional de-duplication (handling emails, URLs, and numerical sentences).
- Discarding pairs that are identical, too long, have mismatched numbers, or contain sentences flagged as non-German/non-English by
pyCLD2. - Mahalanobis Ratio: The novel contribution is the Mahalanobis ratio. They concatenate the sentence vectors from a pair and transform them into a "Mahalanobis space" where the distribution is whitened and decorrelated. In this space, they calculate three vectors: one representing the first sentence's contribution, one for the second, and one for their concatenation. The parallelism score is the ratio of the squared magnitude of the combined vector to the sum of the squared magnitudes of the individual vectors. A higher ratio suggests the pair vector carries more information than its components individually, indicating potential parallelism.
- Post-filtering: After scoring, they perform a final de-duplication step based on seen bigrams to remove redundant pairs.
Datasets
- Training Data: The primary data source was the large, low-precision, web-scraped parallel corpus from the WMT18 shared task (Koehn et al., 2018a). They also utilized existing "clean" parallel corpora (like Europarl and past WMT data) as sources for monolingual data.
- Evaluation Data: The WMT18 shared task provided several test sets (domains like news, medical, IT, etc.) and conditions (10M-word and 100M-word training sets).
- Synthetic Data: For initial testing and sanity checks, they generated synthetic datasets of random normal vectors, some related by a linear transformation, to evaluate the robustness of their Mahalanobis measure.
- Annotated Data: They manually annotated a small set of 300 sentence pairs from the target corpus for sanity checking.
Key Insights and Ideas
- Unsupervised Parallelism is Feasible: The paper demonstrates that effective sentence parallelism can be measured without relying on pre-existing clean parallel data, which is a significant breakthrough for low-resource scenarios.
- Mahalanobis Distance for Anomaly Detection: Applying Mahalanobis distance, a standard tool in anomaly detection, to sentence embeddings offers a principled way to identify "strange" sentence pairs that deviate from expected distributions.
- The "Strange" is Parallel: The core insight is that sentence pairs that are improbable given the distribution of sentence pairs in the corpus are likely to be the truly parallel ones, as opposed to pairs that are simply similar in a superficial way.
- Dimensionality Matters (but not always linearly): Their sanity checks showed that higher dimensionality for sentence vectors generally improved performance, but not always monotonically (e.g., 100D was worse than 50D and 300D). They ultimately settled on 300D.
- Pre- and Post-filtering are Crucial: The effectiveness of their Mahalanobis ratio was significantly boosted by careful pre- and post-filtering steps, highlighting the importance of data cleaning in this task.
Key Results
- Their primary unsupervised submission, NRC-seve-bicov, consistently performed above the median in the 100M-word conditions of the WMT18 Parallel Corpus Filtering task, with many results in the top 10 out of 48 submissions.
- For specific tasks, like translating the European Medicines Agency (EMEA) corpus, their system achieved top rankings even in the 10M-word conditions, outperforming many competitors.
- The Mahalanobis ratio measurement consistently outperformed other unsupervised measurements (linear regression, nonlinear perceptron) on both synthetic and sanity-checking datasets.
- The system achieved 63.4% accuracy in discriminating parallel from non-parallel pairs on their 300-sentence annotated set using 300-dimensional vectors.
Limitations and Future Work
- 10M Condition Performance: The authors acknowledge that their unsupervised systems performed weaker in the 10M-word conditions compared to others, suggesting that while they are good at identifying parallel sentences, they might be less adept at identifying sentences that are useful for MT training beyond simple parallelism.
- Bias Towards Similar Sentences: They suspect their system might have a bias towards picking relatively similar sentences, rather than the more diverse sets needed for MT training.
- Future Work:
- Mitigating the tendency to prefer uninteresting selections of sentences.
- Investigating a word-level analogue of the Mahalanobis ratio measurement, as sentence vectors might not be fine-grained enough to capture all aspects of quality.
Summary updated