Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation

Chenze Shao, Yang Feng

TL;DR

Catastrophic forgetting isn't just a problem for models learning new tasks—it happens within a single training epoch. This "imbalanced training" causes models to favor recent data while forgetting earlier samples, a problem especially severe in low-resource Neural Machine Translation (NMT). The authors propose Complementary Online Knowledge Distillation (COKD), a method that uses dynamic teachers to "remind" the student model of forgotten knowledge, leading to significant gains in translation quality.

What Problem Does This Paper Address?

In the world of AI, "catastrophic forgetting" usually refers to a model losing its ability to perform Task A after being trained on Task B. However, this paper identifies a more insidious version of the problem: imbalanced training within a static dataset.

Because we train neural networks using mini-batch gradient descent, the model sees data sequentially. By the time a model reaches the end of a training epoch, it has "overfit" to the most recent batches and partially "forgotten" the ones it saw at the start. The authors prove this by showing a strong negative correlation between when a batch was seen and its final loss—recent batches have much lower loss than early ones. This is particularly damaging for NMT, where the model must juggle thousands of complex linguistic rules that are easily overwritten by new updates.

Methods

The authors introduce Complementary Online Knowledge Distillation (COKD) to force the model to pay equal attention to all training samples.

  1. Data Splitting: The training set is divided into $n+1$ subsets.
  2. Complementary Roles: While the "student" model trains on subset A, a "teacher" model (trained on a different data order) provides distilled knowledge about subsets B, C, and D. This ensures that while the student is learning new things, the teacher is constantly "re-providing" the knowledge the student might otherwise forget.
  3. Online Updates: Unlike traditional distillation where a teacher is pre-trained and fixed, these teachers are updated "on-the-fly" alongside the student.
  4. Teacher Reinitialization: At the end of every epoch, the teachers are reinitialized with the student's weights. This creates a "two-way" knowledge transfer, ensuring the teachers remain high-quality guides as the student improves.

Datasets

The researchers validated their findings across three distinct domains:

  • Image Classification: CIFAR-10 and CIFAR-100.
  • Text Classification: AG-News.
  • Machine Translation:
    • High-resource: WMT14 English-German (En-De).
  • Low-resource: IWSLT15 English-Vietnamese (En-Vi) and WMT17 English-Turkish (En-Tr).
  • Multilingual: TED bilingual datasets (covering 12 language pairs).

Key Insights and Ideas

  • The Secret of Checkpoint Averaging: A common "trick" in NMT is to average the last few saved models (checkpoints). The authors provide a new explanation for why this works: it acts as a noise-reduction mechanism that smooths out the imbalances created by the specific data order of the final epoch.
  • Complexity Breeds Forgetting: Imbalanced training is almost non-existent in simple tasks like CIFAR-10 (10 classes) but becomes a major bottleneck in CIFAR-100 and NMT. The more complex the "rules" the model must learn, the easier they are to forget.
  • Low-Resource Vulnerability: Low-resource NMT is hit hardest. Without a massive volume of data to provide stability, the model's weights swing wildly toward the most recent samples, leading to poor generalization.
  • Momentum Matters: The "sudden rise" in loss at the very end of an epoch (seen in their charts) is attributed to the momentum factor in optimizers like Adam; the impact of earlier updates isn't fully released until several steps later.

Key Results

  • Significant BLEU Gains: On low-resource tasks (En-Tr and En-Vi), COKD outperformed standard Transformer baselines by ~3.0 BLEU points, a substantial margin in translation research.
  • Better than Baselines: COKD consistently beat other popular techniques like Word-level Knowledge Distillation and Deep Mutual Learning.
  • Quantifiable Balance: Using Spearman correlation to measure imbalance, the authors showed that COKD reduced the correlation coefficient from a severe -0.64 to a much more stable -0.16.
  • TED Bilingual Success: On the TED dataset, COKD improved performance across the board, with an average gain of 1.59 BLEU for English-to-X and 2.15 BLEU for X-to-English directions.

Limitations and Future Work

  • Computational Overhead: Training with teachers in an online fashion increases the computational cost. While the authors recommend using just one teacher ($n=1$) to balance performance and cost, it still roughly doubles the training time compared to a standard baseline.
  • Hyperparameter Tuning: The method relies on a loss weight ($\alpha$) to balance the student's own learning vs. the teacher's distillation. The authors found the model is sensitive to this, generally preferring a very high $\alpha$ (0.95).
  • Scaling to Massive Data: While highly effective for low-resource and medium-resource tasks, the gains on massive datasets (like WMT14 En-De) were smaller, suggesting that "imbalanced training" might be naturally mitigated by sheer data volume. Future work could explore more efficient ways to apply COKD to these "big data" scenarios.

Summary updated

Abstract

Neural networks tend to gradually forget the previously learned knowledge when learning multiple tasks sequentially from dynamic data distributions. This problem is called \textit{catastrophic forgetting}, which is a fundamental challenge in the continual learning of neural networks. In this work, we observe that catastrophic forgetting not only occurs in continual learning but also affects the traditional static training. Neural networks, especially neural machine translation models, suffer from catastrophic forgetting even if they learn from a static training set. To be specific, the final model pays imbalanced attention to training samples, where recently exposed samples attract more attention than earlier samples. The underlying cause is that training samples do not get balanced training in each model update, so we name this problem \textit{imbalanced training}. To alleviate this problem, we propose Complementary Online Knowledge Distillation (COKD), which uses dynamically updated teacher models trained on specific data orders to iteratively provide complementary knowledge to the student model. Experimental results on multiple machine translation tasks show that our method successfully alleviates the problem of imbalanced training and achieves substantial improvements over strong baseline systems.
BibTeX
@inproceedings{shao2022overcoming,
  author = {Chenze Shao and Yang Feng},
  title = {{Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=7McC3GmnUI},
}
Figure