Improving the Quality of Web-mined Parallel Corpora of Low-Resource Languages using Debiasing Heuristics

Aloka Fernando, Nisansa de Silva, Menan Velayuthan, Charitha Rathnayake, Surangika RanathungaEMNLP 2025

TL;DR

Web-mined parallel corpora are crucial for training low-resource language NMT models, but they are often noisy. This paper introduces a set of rule-based heuristics to filter these corpora, significantly improving NMT performance and reducing the disparity observed when using different pre-trained multilingual language models (multiPLMs) for ranking.

What Problem Does This Paper Address?

Training Neural Machine Translation (NMT) models, especially for low-resource languages (LRLs), heavily relies on large parallel corpora. While web-mining offers a vast source of such data (e.g., CCAligned, CCMatrix), these corpora are inherently noisy. A common approach to mitigate this noise is Parallel Data Curation (PDC), which typically involves ranking sentence pairs based on similarity scores derived from multiPLM embeddings. However, previous work has shown that the choice of multiPLM significantly impacts the quality of the filtered corpus and, consequently, the performance of NMT models trained on it. This disparity suggests that different multiPLMs might be biased towards certain types of sentence pairs, leading to the inclusion of noise that hinders NMT performance. This paper aims to address this by systematically investigating how rule-based heuristics can debias these corpora and reduce the multiPLM-induced disparities.

Methods

The core of this work lies in applying and evaluating a suite of rule-based heuristics for filtering noisy parallel sentences. The authors extend an existing taxonomy of noise categories and introduce a new category, CCN (high-overlapping non-translatable text), to better capture noise like excessive numbers, acronyms, and URLs.

The heuristics explored fall into several categories:

  • Deduplication-based: Removing identical or near-identical sentence pairs, including n-gram based deduplication.
  • Length-based (sLength): Filtering out sentences that are too short to provide sufficient linguistic information.
  • LID-based (Language Identification): Removing sentences that are not in the expected language or are untranslated.
  • Ratio-based: Identifying sentences with significant structural imbalances between source and target (e.g., sentence length ratio, word-to-alpha ratio).

The paper systematically evaluates these heuristics individually and in combination, assessing their impact on NMT performance (measured by ChrF++ scores) and the disparity between NMT models trained with corpora filtered by different multiPLMs (LASER3, XLM-R, LaBSE). They also conduct human evaluations to quantify the noise reduction achieved by these heuristics.

Datasets

The study utilizes two publicly available web-mined parallel corpora:

  • CCMatrix: Extracted using LASER2-based sentence embeddings.
  • CCAligned: Extracted using document-level and sentence-level alignment based on multilingual embeddings.

These corpora were used for the language pairs: English-Sinhala (En-Si), English-Tamil (En-Ta), and Sinhala-Tamil (Si-Ta). For NMT experiments, the dev and devtest subsets from the Flores-200 dataset were used for validation and evaluation.

Key Insights and Ideas

  • MultiPLM Bias is Real and Impacts NMT: The observed disparity in NMT performance across different multiPLMs is directly linked to their inherent biases in ranking sentence pairs, leading to the inclusion of noisy data.
  • Heuristics are a Simple Yet Effective Debiasing Tool: Rule-based heuristics, when applied systematically, can effectively filter out noise that deep learning-based methods might miss or even amplify due to multiPLM biases.
  • No Single Heuristic is a Silver Bullet: Different heuristics excel at addressing specific types of noise. The most effective approach involves combining multiple heuristics.
  • Combined Heuristics Significantly Reduce Disparity: The best-performing heuristic combinations not only boost NMT scores but also dramatically reduce the performance gap between models trained on corpora filtered by different multiPLMs.
  • Human Evaluation Confirms Heuristic Effectiveness: Human annotations show a substantial reduction in noise after heuristic filtering, with error percentages dropping significantly.

Key Results

  • NMT Performance Improvement: Applying heuristics, especially combined ones, consistently led to higher ChrF++ scores compared to baseline NMT models trained on unfiltered top-ranked sentences. For instance, combined heuristics achieved gains of up to 18.63 points for CCMatrix-EnSi.
  • Disparity Reduction: The disparity in NMT scores between models using different multiPLMs was significantly reduced. For example, the disparity between LASER3 and XLM-R/LaBSE was reduced by an average of 95.87% for certain corpora after applying combined heuristics.
  • Best Performing Heuristic Combination: The most effective combination identified was dedup+punctNums+(n)gram+sLength+LIDThresh+sentWRatio, with the specific n-gram value varying based on the dataset.
  • Noise Reduction: Human evaluation indicated that after heuristic filtering, the average error percentages for LASER, XLM-R, and LaBSE dropped to 2.17%, 2.50%, and 1.0%, respectively.

Limitations and Future Work

  • LID Suboptimality for Si/Ta: The language identification (LID) model used was suboptimal for Sinhala and Tamil, suggesting a need for improved LID models for these languages.
  • Limited Language Coverage: The study was limited to three language pairs due to the availability of human annotators. Expanding to more LRLs would require more annotators.
  • Residual Noise: Even after heuristic filtering, some noise (primarily untranslated text and overlapping non-translatable text) remains. Future work could involve alignment models or translation post-editing to address this.
  • Ethical Considerations: The authors acknowledge the challenge of manually checking large datasets for offensive content and ensure fair compensation and privacy for human annotators.

Summary updated

BibTeX
@inproceedings{fernando2025improving,
  author = {Aloka Fernando and Nisansa de Silva and Menan Velayuthan and Charitha Rathnayake and Surangika Ranathunga},
  title = {{Improving the Quality of Web-mined Parallel Corpora of Low-Resource Languages using Debiasing Heuristics}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure