Abstract
Neural networks tend to gradually forget the previously learned knowledge when learning multiple tasks sequentially from dynamic data distributions. This problem is called catastrophic forgetting, which is a fundamental challenge in the continual learning of neural networks. In this work, we observe that catastrophic forgetting not only occurs in continual learning but also affects the traditional static training. Neural networks, especially neural machine translation models, suffer from catastrophic forgetting even if they learn from a static training set. To be specific, the final model pays imbalanced attention to training samples, where recently exposed samples attract more attention than earlier samples. The underlying cause is that training samples do not get balanced training in each model update, so we name this problem imbalanced training. To alleviate this problem, we propose Complementary Online Knowledge Distillation (COKD), which uses dynamically updated teacher models trained on specific data orders to iteratively provide complementary knowledge to the student model. Experimental results on multiple machine translation tasks show that our method successfully alleviates the problem of imbalanced training and achieves substantial improvements over strong baseline systems.
Introduction
In the field of artificial intelligence, catastrophic forgetting is traditionally defined as a model's loss of ability to perform an earlier task after being trained on a new one. However, research presented at ACL 2022 challenges this view, demonstrating that forgetting occurs on a much smaller scale during every single training epoch of a static dataset. This phenomenon, termed imbalanced training, is particularly prevalent in complex tasks like Neural Machine Translation (NMT).
The Problem: Imbalanced Training
Standard training utilizes mini-batch gradient descent, where a model sees data sequentially. By the time an epoch concludes, the model has undergone thousands of updates since it last processed the initial batches. This leads to a bias: the final model parameters are more optimized for recently seen samples (resulting in lower loss) while partially "forgetting" the patterns learned from earlier samples (resulting in higher loss).
As shown in the figure above, the severity of this imbalance correlates with task complexity. While simple tasks like CIFAR-10 show almost no correlation between batch order and loss, complex NMT tasks like English-Turkish (WMT17-EnTr) exhibit a strong negative correlation (corr=-0.64), indicating significant forgetting of early-epoch data.
Proposed Method: Complementary Online Knowledge Distillation (COKD)
To mitigate imbalanced training, the authors propose Complementary Online Knowledge Distillation (COKD). This framework trains a "student" model and one or more "teacher" models simultaneously from scratch using the following mechanism:
- Data Splitting and Offset Ordering: The training set is divided into subsets. The student and teachers are fed these subsets in different, complementary orders. For example, while the student learns from the first subset, a teacher might be processing the final subset.
- Knowledge Distillation: The student is trained to minimize its own loss while simultaneously mimicking the probability distributions (soft targets) of the teachers. Because teachers are looking at different data, they act as a "memory" for the parts of the dataset the student is currently neglecting.
- Iterative Reinitialization: At the end of each epoch, teachers are reinitialized with the student’s parameters. This ensures the models do not diverge too far and maintains a productive two-way transfer of knowledge.
Experimental Results
The COKD method was validated across image classification, text classification, and various NMT tasks, showing consistent improvements:
- Low-Resource NMT: On the WMT17 English-Turkish task, COKD improved the BLEU score from 13.42 to 16.66 (+3.24). On English-Vietnamese, it rose from 29.08 to 31.95 (+2.87).
- Reduced Imbalance: The Spearman correlation coefficient for the Turkish task dropped from -0.64 to -0.16, indicating a much more balanced learning process.
- Multilingual Robustness: The method showed gains across all 12 language pairs in the TED bilingual dataset.
Discussion and Insights
This research provides a theoretical explanation for why checkpoint averaging—a common practice in NMT where the last few saved versions of a model are averaged—is effective. Each checkpoint is biased toward different batches at the end of the epoch; averaging them effectively cancels out the noise of imbalanced training.
Furthermore, the study highlights that low-resource NMT models are particularly vulnerable to this effect. These models often "chase their tails" within an epoch, forgetting earlier rules as quickly as they learn new ones. COKD provides the necessary stability to break this cycle.
Conclusion
By identifying catastrophic forgetting within static training, this work opens new avenues for optimizing neural network training. COKD offers a robust solution for imbalanced training, particularly for complex, low-resource tasks where data efficiency is paramount. While it introduces some computational overhead, the significant gains in translation quality and model stability make it a compelling framework for modern NMT systems.
Summary updated