TL;DR
This paper details the extensive human evaluation efforts undertaken for the ParaCrawl project, a massive collection of web-crawled parallel data for machine translation. It highlights the challenges of ensuring data quality and presents a multi-faceted approach combining error annotation, post-editing, and search-based evaluation to identify areas for pipeline improvement.
What Problem Does This Paper Address?
Neural Machine Translation (NMT) systems are data-hungry, but high-quality parallel data (text in two languages that are translations of each other) is often scarce, varied, or of poor quality. Web crawling offers a way to gather vast amounts of parallel data, but this data is inherently noisy and prone to errors. The ParaCrawl project aimed to create a large, usable parallel corpus from the web, but faced the critical challenge of assessing and improving the quality of this data to ensure its effectiveness for NMT. This paper addresses how they tackled this challenge through rigorous human evaluation.
Methods
The authors employed a three-pronged human evaluation strategy:
- Error Annotation-based Evaluation: This involved annotators labeling parallel sentences based on a hierarchical error typology. Categories included wrong language identification, incorrect alignment, wrong tokenization, machine translation artifacts, general translation errors (lexical and syntactic), and "free translation" (content reformulated for editorial purposes). This method aimed to pinpoint specific types of errors within the corpus.
- Post-Editing (PE)-based Evaluation: Here, professional translators were tasked with post-editing the output of NMT systems trained on ParaCrawl data. This evaluated the practical utility of the corpora by observing how much effort was required to correct MT output. They also used this to compare baseline MT systems against those augmented with ParaCrawl data.
- Search-based Evaluation: This involved using a tool called Corset to allow professional translators to perform searches over the parallel corpora. The goal was to assess if the corpora could serve as a resource for tasks beyond MT training, such as finding translation equivalents or understanding usage.
Additionally, they conducted extrinsic evaluation through machine translation experiments, comparing NMT models trained on different versions of the ParaCrawl corpus against baseline models.
Datasets
The primary dataset discussed is the ParaCrawl corpus, which evolved through several versions (3, 6, 7, 8, and 9). The paper also mentions using data from the WMT (Workshop on Machine Translation) shared tasks for training and evaluation, specifically WMT16 and WMT17 datasets. A TED Talks test set was also used for extrinsic evaluation.
Key Insights and Ideas
- Human evaluation is crucial for web-crawled data: Despite advancements in automatic metrics, human evaluation remains indispensable for understanding the nuances of data quality and identifying actionable insights for pipeline improvement.
- Error typology needs careful application: The hierarchical error typology, while useful, proved challenging in practice, with annotators sometimes misclassifying errors (e.g., using "MT error" for "Bad Alignment"). This highlights the need for clear guidelines and ongoing refinement.
- Post-editing reveals practical utility: PE evaluation showed that MT systems trained with ParaCrawl data generally produced better output, requiring less editing, confirming the corpus's value. However, it also revealed discrepancies with automatic metrics in some cases.
- Web-crawled data can have diverse uses: The search-based evaluation indicated that the corpora could be valuable for translators beyond just MT training, offering a resource for finding translation examples and understanding language usage.
- Quality is not just about quantity: While ParaCrawl aimed for scale, the evaluations consistently pointed to the importance of data quality and the need to refine processing pipelines to minimize errors.
Key Results
- Error Annotation: Initial campaigns revealed issues with language identification, sentence segmentation, and encoding. Later campaigns showed low inter-annotator agreement, indicating challenges in consistently applying the error typology.
- Post-Editing: In PE evaluations, MT systems augmented with ParaCrawl data were consistently preferred over baseline systems. Edit distances generally confirmed this, showing closer proximity to the final translation. However, for English-Czech, automatic metrics showed improvement, but PE results did not, suggesting limitations in the test sets or the automatic metrics themselves.
- WMT-based Evaluation: BLEU scores generally improved with the addition of ParaCrawl data, though some versions showed conflicting results with other metrics like chrF and COMET.
Limitations and Future Work
- Limitations:
- The authors acknowledge uncertainty about whether improvements were due to data quantity or quality.
- They question the suitability of the chosen test sets for representing the overall quality of the corpora, especially given the wide range of languages.
- The consistency of error annotation was a significant challenge, with low inter-annotator agreement in some campaigns.
- The scope of the PE and WMT-based evaluations (limited language pairs) might not fully represent the quality of the entire corpus.
- Future Work:
- The insights gained are being used to inform the MaCoCu project, which aims to build even larger multilingual corpora.
- Future human evaluation may focus on simpler, single-issue tasks rather than complex hierarchical ones.
- Exploring other evaluation tasks like direct assessment, ranking, and fluency is suggested.
- For automatic evaluation, the paper suggests considering more balanced corpora sizes and exploring fine-tuning techniques beyond simple concatenation.
Summary updated