Zipporah: a Fast and Scalable Data Cleaning System for Noisy Web-Crawled Parallel Corpora

Hainan Xu, Philipp KoehnEMNLP 2017 · OpenReview

Abstract

Zipporah is a fast, scalable system designed to clean noisy, web-crawled parallel corpora for machine translation. By training a simple logistic regression model on "adequacy" and "fluency" features—using synthetically generated noise as negative examples—it can identify high-quality sentence pairs so effectively that using just a fraction of a cleaned dataset can outperform using the entire noisy corpus by significant margins.

Introduction

Statistical Machine Translation (SMT) and modern neural translation models are highly sensitive to data quality. While the internet provides vast amounts of parallel text, web-crawled data is often riddled with machine-translated gibberish, mismatched sentences, or monolingual pairs. Existing cleaning tools are frequently too slow for billion-word datasets or focus too narrowly on domain adaptation rather than general quality. Zipporah addresses these challenges by providing a computationally efficient system capable of distinguishing "good" translations from "bad" ones at scale.

Methodology

Zipporah employs a classification-based approach, mapping every sentence pair into a 2D feature space defined by adequacy and fluency.

Feature Engineering

  1. Adequacy: This metric assesses whether the source and target sentences share the same meaning. Zipporah uses a "bag-of-words" translation model to calculate the cross-entropy between the word distribution of the source sentence (translated via a dictionary) and the actual target sentence.
  2. Fluency: This measures how natural the target sentence sounds using N-gram language models. The score is derived from the ratio of the sentence's negative log-likelihood to its length.

Synthetic Noise and Classification

To train the classifier without manual labeling, the authors generate synthetic "bad" data from a clean corpus:

  • Type A: Shuffled sentences (Good fluency, poor adequacy).
  • Type B: Shuffled words (Poor fluency, good adequacy).
  • Type C: Both (Poor fluency and adequacy).

A logistic regression model is trained to separate clean data from this synthetic noise. To capture non-linear relationships, features are mapped into higher-order forms.

Figure 1: Distribution of good and bad sentence pairs in the fluency-adequacy feature space.

Experimental Setup

The system was evaluated using early versions of ParaCrawl for French-English (340M words), German-English (487M words), and Spanish-English (70M words). Europarl served as the clean reference for building dictionaries and language models. Performance was measured across diverse domains including newstest2011, TED-talks, and movie subtitles.

Results and Discussion

Zipporah consistently outperformed random selection and matched or exceeded the performance of more complex, less scalable systems like qe-clean.

The "Less is More" Phenomenon

A key finding is that adding low-quality data can dilute model parameters and degrade performance. Zipporah allows researchers to find the "sweet spot" where quality and quantity are balanced. For instance, on the German-English TED-talks dataset, Zipporah achieved a 5.5 BLEU score improvement over a random baseline by selecting less than 2% of the available data.

Figure 3: BLEU scores vs. number of words for German-English data, showing Zipporah's performance peak compared to baselines.

Scalability and Efficiency

By utilizing a bag-of-words approach instead of expensive word alignments, Zipporah is significantly faster than previous methods. This efficiency makes it suitable for the massive datasets typical of modern web crawls.

Conclusion

Zipporah demonstrates that effective data cleaning is as much about identifying what to exclude as what to include. By leveraging synthetic noise and simple yet robust features, it provides a scalable solution for building high-quality parallel corpora. Future work may involve considering corpus diversity and reducing dependence on pre-existing dictionaries for adequacy scoring.

Summary updated

BibTeX
@inproceedings{xu2017zipporah,
  author = {Hainan Xu and Philipp Koehn},
  title = {{Zipporah: a Fast and Scalable Data Cleaning System for Noisy Web-Crawled Parallel Corpora}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=HkZF2bM_bB},
}
Figure