TL;DR
This paper presents a hybrid pipeline combining rule-based filtering and machine learning to clean noisy web-crawled parallel corpora for machine translation training. The core innovation is a logistic regression model trained on automatically annotated data, which effectively identifies high-quality translation units while respecting task constraints on external data and alignment information.
What Problem Does This Paper Address?
Training robust machine translation (MT) systems requires large amounts of high-quality parallel text (sentences and their translations). However, web-crawled data, while abundant, is often noisy, containing misalignments, errors, and irrelevant content. The "Parallel Corpus Filtering" task specifically challenged participants to extract clean subsets of English-German parallel text from a massive, noisy corpus. This paper tackles the problem of efficiently and effectively filtering this noisy web data to produce usable training sets for MT, a task analogous to translation memory cleaning and quality estimation.
Methods
The authors propose a three-stage hybrid pipeline:
- Filtering Rules: This initial stage removes obviously problematic translation units. Rules include discarding units that are too short, have mismatched language codes (as detected by Cybozu), or have a low-quality alignment score (based on the Church-Gale score). This step significantly reduces the corpus size from over 104 million to about 11 million units.
- Machine Learning: This is the core of the pipeline. A logistic regression classifier is trained to predict the probability that a translation unit is of high quality. Crucially, due to task restrictions, the training data is generated automatically. This is done by using the Hunalign tool's score: units with a score above a threshold (0.92) are treated as positive examples, and those below as negative. The model uses a rich set of features:
- Presence/Absence Features: Detects the presence of specific entities (tags, URLs, emails, names, numbers, punctuation, capital letters) in both source and target segments.
- Entity Similarity Features: Computes cosine similarity for these entities between source and target.
- Capital Letters Features: Measures the difference in words containing capital letters.
- Alignment Features: Includes language code agreement and a modified Gale-Church score, which correlates segment lengths.
- Fluency Features: Uses perplexity scores from KenLM language models trained on cleaned English and German corpora to assess segment fluency.
- Re-ranking Rules: A final set of rules is applied to re-score the probability outputs from the machine learning module. These rules penalize units that violate specific patterns, such as:
- Same Digits Rule: Ensures digits in the source appear in the target (allowing for different date formats).
- Same Numbers Rule: Similar to the digits rule, but for general numbers.
- Rule URL: Penalizes units where URLs are disproportionately long compared to the normal text.
- Rule Tags: Ensures identical tags appear in both segments. Finally, a cosine similarity measure is used on English segments to ensure diversity in the selected subsets.
Datasets
- Training Data: The paper uses a large web-crawled English-German parallel corpus. The training set for the logistic regression model is generated automatically from this corpus using the Hunalign tool's scores.
- Evaluation Data: The paper describes manual annotation of a subset of the automatically annotated training data for evaluation purposes. The primary evaluation of the system's performance was conducted by the task organizers, who used BLEU scores from MT systems trained on the filtered subsets.
Key Insights and Ideas
- Automatic Annotation Effectiveness: The paper demonstrates that a heuristic-based automatic annotation strategy (using Hunalign scores) can yield a surprisingly good training set for a binary classifier, achieving a balanced accuracy of 0.83 on a manually annotated gold standard. This is crucial given the task's restriction against using external parallel corpora for training.
- Hybrid Approach Synergy: Combining rule-based filtering with machine learning proves effective. Rules handle obvious errors efficiently, while ML captures more nuanced quality signals. The re-ranking rules further refine the output.
- Feature Engineering for Quality: The diverse set of features, ranging from simple entity presence to language model perplexity and alignment scores, highlights the complexity of defining "translation quality" and the need for multi-faceted approaches.
- Diversity Matters: The inclusion of a cosine similarity-based diversity enhancement step suggests that simply maximizing quality scores might lead to redundant or overly similar translation units, which is undesirable for MT training.
Key Results
- The automatic annotation heuristic achieved a balanced accuracy of 0.83 when compared against a manually annotated gold standard.
- The initial filtering step reduced the corpus from 104,002,521 units to 11,030,014 units.
- The system's performance in the shared task placed it in the range of 22-31 out of 48 submissions, depending on the evaluation subset and MT system used.
Limitations and Future Work
- Representativeness of Auto-Annotation: The authors acknowledge that the automatically annotated set, while good, might not be a perfectly representative sample of the entire test set. A larger manually annotated set would be ideal for a more robust evaluation of the auto-annotation strategy.
- Bible Passages: The manual annotation revealed a significant number of Bible passages, which have unique linguistic characteristics that might not generalize well to other text types. While useful for MT in this task, they might be problematic for general-purpose MT.
- Computational Constraints: The paper mentions that computing cosine similarity matrices for diversity enhancement was restricted to 30,000 lines due to server limitations, suggesting potential scalability issues for extremely large datasets.
- Future Work: The authors make their core system, the manually annotated set, and the evaluation script publicly available on GitHub, facilitating further research and development in this area.
Summary updated