Introduction
Neural Machine Translation (NMT) has seen significant advances, yet it remains highly dependent on massive datasets. However, not all data points in these large-scale corpora contribute equally to model performance. This paper identifies a category of "inactive" examples—data that is noisy, overly complex, or redundant—which can actually impede training. To address this, the authors propose Data Rejuvenation, a framework designed to transform these problematic examples into useful training signals through a process of identification and re-labeling.
Methodology: The Data Rejuvenation Framework
The proposed framework consists of a three-phase pipeline aimed at cleaning and simplifying the training data.
- Identification: An initial NMT model is trained on the full raw dataset. The "sentence-level output probability" is calculated for every example. Sentences falling into the lowest probability bracket (typically the bottom 10%) are flagged as inactive, while the remainder are considered active.
- Rejuvenation: A separate "rejuvenation model" is trained using only the high-quality active examples. This model then performs forward-translation on the source side of the inactive examples, replacing the original, potentially noisy target sentences with cleaner, model-generated alternatives.
- Final Training: The final NMT model is trained from scratch on a merged dataset consisting of the original active examples and the newly rejuvenated synthetic examples.
This process functions as a targeted form of knowledge distillation, leveraging the model's own confidence to simplify the most challenging portions of the dataset.
Experimental Setup
The framework was evaluated on two major benchmarks: WMT14 English-German (En-De) (4.5M pairs) and WMT14 English-French (En-Fr) (35.5M pairs). The authors tested the method across diverse architectures, including LSTMs, the standard Transformer, and DynamicConv, to ensure broad applicability.
Key Insights into Data Inactivity
A central finding of the study is that "inactivity" is primarily a property of the data itself rather than the model architecture. There was an 80% overlap in flagged inactive sentences across different model types and random seeds. Furthermore, inactive examples are not merely "noise"; they often consist of "source-translated" text—human translations originally written from target-to-source—which can introduce unnatural phrasing for a forward-translation model. By re-labeling these examples, the rejuvenation model provides targets with lower uncertainty and better coverage, making them easier for the final model to learn.
Results and Performance
Data Rejuvenation consistently yielded improvements in translation quality. On the En-De task, it provided a +0.8 BLEU point boost for both Transformer-Base and Transformer-Big models.
Beyond raw performance, the method improved training stability, resulting in faster convergence and fewer loss fluctuations. Interestingly, the researchers found that simply removing the 10% most inactive examples often maintained or even slightly improved performance, confirming that a significant portion of standard datasets can act as "dead weight." When combined with other techniques like data diversification, the benefits were additive, reaching up to +1.1 BLEU over the baseline.
Conclusion and Future Work
While Data Rejuvenation offers clear performance gains, it introduces additional computational overhead due to its multi-phase nature. Future research may focus on streamlining this pipeline or applying the concept to other sequence-to-sequence tasks such as text summarization or dialogue generation. Despite the added training time, the framework provides a robust method for maximizing the utility of large-scale, imperfect datasets.
Summary updated