Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation

Wenxiang Jiao, Xing Wang, Shilin He, Irwin King, Michael R. Lyu, Zhaopeng Tu

Introduction

Neural Machine Translation (NMT) has seen significant advances, yet it remains highly dependent on massive datasets. However, not all data points in these large-scale corpora contribute equally to model performance. This paper identifies a category of "inactive" examples—data that is noisy, overly complex, or redundant—which can actually impede training. To address this, the authors propose Data Rejuvenation, a framework designed to transform these problematic examples into useful training signals through a process of identification and re-labeling.

Methodology: The Data Rejuvenation Framework

The proposed framework consists of a three-phase pipeline aimed at cleaning and simplifying the training data.

Overview of the Data Rejuvenation framework, illustrating the identification of inactive examples and their subsequent rejuvenation via a specialized model.
  1. Identification: An initial NMT model is trained on the full raw dataset. The "sentence-level output probability" is calculated for every example. Sentences falling into the lowest probability bracket (typically the bottom 10%) are flagged as inactive, while the remainder are considered active.
  2. Rejuvenation: A separate "rejuvenation model" is trained using only the high-quality active examples. This model then performs forward-translation on the source side of the inactive examples, replacing the original, potentially noisy target sentences with cleaner, model-generated alternatives.
  3. Final Training: The final NMT model is trained from scratch on a merged dataset consisting of the original active examples and the newly rejuvenated synthetic examples.

This process functions as a targeted form of knowledge distillation, leveraging the model's own confidence to simplify the most challenging portions of the dataset.

Experimental Setup

The framework was evaluated on two major benchmarks: WMT14 English-German (En-De) (4.5M pairs) and WMT14 English-French (En-Fr) (35.5M pairs). The authors tested the method across diverse architectures, including LSTMs, the standard Transformer, and DynamicConv, to ensure broad applicability.

Key Insights into Data Inactivity

A central finding of the study is that "inactivity" is primarily a property of the data itself rather than the model architecture. There was an 80% overlap in flagged inactive sentences across different model types and random seeds. Furthermore, inactive examples are not merely "noise"; they often consist of "source-translated" text—human translations originally written from target-to-source—which can introduce unnatural phrasing for a forward-translation model. By re-labeling these examples, the rejuvenation model provides targets with lower uncertainty and better coverage, making them easier for the final model to learn.

Results and Performance

Data Rejuvenation consistently yielded improvements in translation quality. On the En-De task, it provided a +0.8 BLEU point boost for both Transformer-Base and Transformer-Big models.

Comparison of BLEU scores with and without rejuvenation across different ratios of inactive examples. Rejuvenation consistently outperforms the baseline and maintains performance even as more data is processed.

Beyond raw performance, the method improved training stability, resulting in faster convergence and fewer loss fluctuations. Interestingly, the researchers found that simply removing the 10% most inactive examples often maintained or even slightly improved performance, confirming that a significant portion of standard datasets can act as "dead weight." When combined with other techniques like data diversification, the benefits were additive, reaching up to +1.1 BLEU over the baseline.

Conclusion and Future Work

While Data Rejuvenation offers clear performance gains, it introduces additional computational overhead due to its multi-phase nature. Future research may focus on streamlining this pipeline or applying the concept to other sequence-to-sequence tasks such as text summarization or dialogue generation. Despite the added training time, the framework provides a robust method for maximizing the utility of large-scale, imperfect datasets.

Summary updated

Abstract

Large-scale training datasets lie at the core of the recent success of neural machine translation (NMT) models. However, the complex patterns and potential noises in the large-scale data make training NMT models difficult. In this work, we explore to identify the inactive training examples which contribute less to the model performance, and show that the existence of inactive examples depends on the data distribution. We further introduce data rejuvenation to improve the training of NMT models on large-scale datasets by exploiting inactive examples. The proposed framework consists of three phases. First, we train an identification model on the original training data, and use it to distinguish inactive examples and active examples by their sentence-level output probabilities. Then, we train a rejuvenation model on the active examples, which is used to re-label the inactive examples with forward-translation. Finally, the rejuvenated examples and the active examples are combined to train the final NMT model. Experimental results on WMT14 English-German and English-French datasets show that the proposed data rejuvenation consistently and significantly improves performance for several strong NMT models. Extensive analyses reveal that our approach stabilizes and accelerates the training process of NMT models, resulting in final models with better generalization capability.
BibTeX
@inproceedings{jiao2020data,
  author = {Wenxiang Jiao and Xing Wang and Shilin He and Irwin King and Michael R. Lyu and Zhaopeng Tu},
  title = {{Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.176},
}
Figure