Extracting In-domain Training Corpora for Neural Machine Translation Using Data Selection Methods

Catarina Cruz Silva, Chao-Hong Liu, Alberto Poncelas, Andy WayEMNLP 2018

TL;DR

This paper investigates how data selection methods, previously successful for Statistical Machine Translation (SMT), impact Neural Machine Translation (NMT). They found that while these methods do improve NMT performance, the gains are less pronounced than in SMT, with Feature Decay Algorithms (FDA) showing the most promise. The study also highlights the importance of balancing selected data with an initial seed for optimal results.

What Problem Does This Paper Address?

Training effective Neural Machine Translation (NMT) systems often requires massive parallel datasets. However, for specific domains (e.g., legal or medical texts), using a general-purpose corpus can lead to suboptimal performance due to differences in vocabulary and sentence structure. Data selection aims to address this by identifying and extracting the most relevant subsets of data from large corpora. This can reduce training time and resources, and crucially, tailor NMT models to perform better on in-domain tasks. While data selection has shown significant benefits for older SMT systems, its effectiveness and optimal application for modern NMT architectures remain an open question.

Methods

The authors explore three established data selection techniques:

  • Cross Entropy Difference (CED): This method ranks sentences based on how well they fit both an out-of-domain language model and an in-domain language model. It leverages perplexity scores to identify sentences that are more specific to the target domain.
  • Term Frequency-Inverse Document Frequency (TF-IDF): A classic information retrieval technique, TF-IDF is adapted here to score sentences based on the importance of terms within them relative to their frequency across the entire corpus.
  • Feature Decay Algorithms (FDA): This approach focuses on extracting sentences that best represent a "seed" set of domain-specific data. It iteratively selects sentences, decaying the influence of features (n-grams) that have already been well-represented, thus promoting diversity and coverage of domain-specific language.

For their NMT experiments, they utilize the Marian framework, employing an attention-based encoder-decoder architecture. Preprocessing involves standard steps like tokenization, truecasing, and Byte-Pair Encoding (BPE). A key aspect of their experimental setup is the preparation of an initial "seed" dataset by cleaning and filtering domain-specific data, and then using this seed in conjunction with the selected data for training. They also investigate the impact of balancing the seed and selected data, finding that oversampling the seed to match the size of the selected data yields better results.

Datasets

The experiments use English-French parallel data from two specific domains:

  • EMEA: Medical data.
  • DGT: Legal data.

These domain-specific corpora are used to create the initial "seed" datasets. For training generic models and as a source for data selection, they use the MultiUN corpus, which consists of news data. They also conduct experiments for English-Spanish translation to increase confidence in their findings.

Key Insights and Ideas

  • Data selection helps NMT, but less than SMT: While data selection methods do improve NMT performance, the gains are not as substantial as those observed in previous studies on SMT. This suggests that NMT architectures might be inherently better at handling broader data distributions or that the selected data is not as critical for NMT's success.
  • FDA emerges as the top performer: Among the three methods tested, Feature Decay Algorithms (FDA) consistently yielded the best results in terms of BLEU scores and Translation Error Rate (TER) for NMT. This indicates its effectiveness in capturing domain-specific nuances for modern neural models.
  • Balancing is crucial: Combining the initial seed data with the selected data by oversampling the seed to match the size of the selected data significantly boosted performance. This highlights the importance of retaining some general domain knowledge from the seed while incorporating specialized content.
  • Smaller seeds can amplify gains: While larger seeds provide more initial information, smaller seeds, when combined with data selection, can lead to larger relative improvements in BLEU scores. This suggests that data selection is particularly impactful when the initial training data is limited.
  • Fine-tuning with selected data can be detrimental: In a fine-tuning scenario, where a model is first trained on generic data, using selected domain data for further training sometimes resulted in lower performance compared to using just the seed data. This might be due to the limited domain-specific vocabulary in the generic corpus or issues with subword tokenization.

Key Results

  • FDA outperforms CED and TF-IDF: In full training scenarios for EMEA EN-FR, FDA achieved the highest BLEU scores, followed by TF-IDF, with CED showing the lowest performance. For example, with a 2M slice of selected data, FDA achieved a BLEU score of .488, compared to .479 for CED.
  • Balanced data is superior: Experiments showed that balancing the seed and selected data (e.g., using a 1:1 ratio) consistently outperformed simply concatenating them.
  • Significant gains with smaller seeds: For the EMEA EN-FR domain, using a 50K seed with data selection resulted in BLEU score increases of 16-18 points compared to the baseline, whereas a larger seed (240K) yielded gains of around 5-9 points.
  • Human evaluation confirms benefits: Human evaluation of Fluency and Adequacy showed that models trained with selected data consistently outperformed those trained with the whole corpus, especially when fine-tuning was applied. For EMEA EN-FR, Adequacy scores improved from 4.01 (seed only) to 4.18 (seed + best slice).

Limitations and Future Work

  • Limited domain data for fine-tuning: The authors suggest that the negative impact of data selection in fine-tuning might be due to the MultiUN data containing very little domain-specific information, and that fine-tuned models might benefit more from access to a richer generic vocabulary.
  • Comparison with human evaluation metrics: The paper proposes future work to make more comparisons between human evaluation metrics (Adequacy, Fluency) and standard MT performance metrics (BLEU, TER).
  • Exploring other data selection methods: While three methods were explored, there are many other data selection techniques that could be investigated for NMT.
  • Impact on different NMT architectures: The study focused on a specific attention-based encoder-decoder architecture. Investigating the impact of data selection on other NMT models (e.g., Transformers) would be valuable.

Summary updated

BibTeX
@inproceedings{silva2018extracting,
  author = {Catarina Cruz Silva and Chao-Hong Liu and Alberto Poncelas and Andy Way},
  title = {{Extracting In-domain Training Corpora for Neural Machine Translation Using Data Selection Methods}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure