Mitigating Catastrophic Forgetting in Continual Learning through Model Growth

Ege Süalp, Mina RezaeiarXiv 2025 · DOI · arXiv

Abstract: This study investigates the relationship between Large Language Model (LLM) architectural initialization and catastrophic forgetting (CF). By comparing a standard 7B parameter model to "StackLLM"—a model grown by stacking layers of a smaller parent model—the researchers evaluate how model growth affects knowledge retention during sequential fine-tuning. The findings suggest that while stacking provides a modest buffer against forgetting in reasoning and reading comprehension, it introduces a unique challenge: increased resistance to the de-biasing effects typically observed during instruction tuning.

The Challenge of Catastrophic Forgetting

Catastrophic Forgetting (CF) remains a primary obstacle in the development of versatile LLMs. When a model is fine-tuned on a specific new task, it frequently loses the general capabilities or domain knowledge acquired during its initial pre-training. As models scale, training from scratch becomes prohibitively expensive, leading to the rise of "model growth" strategies. These methods, such as layer stacking, can reduce compute costs by 25-50%. This research explores whether the better initialization provided by a smaller "parent" model helps the resulting "child" model maintain its knowledge base more effectively.

Methodology: Stacking vs. Scratch

The researchers compared two 7B parameter models, both trained on 300 billion tokens:

  1. Baseline LLM: A standard decoder-only model trained from scratch.
  2. StackLLM: A model that began at 1/4th the target depth, trained on 10B tokens, and was then grown to the full 7B size via layer duplication before completing the 300B token training.

Both models underwent Continual Fine-tuning across three sequential tasks without data rehearsal: Text Simplification (Simp), Empathetic Dialogue Generation (Emdg), and Inquisitive Question Generation (InqQG).

Training Dynamics and the Staircase Effect

During the fine-tuning process, the authors observed a distinct "staircase pattern" in the training loss. Sharp drops at the beginning of epochs were often followed by rising validation loss, indicating a high susceptibility to overfitting. This necessitated the use of cosine learning rate schedulers and weight decay to maintain model stability.

Training and evaluation loss curves showing the 'staircase' effect and overfitting during sequential fine-tuning.

Key Results: A Shield Against Forgetting

The study utilized the Forgetting Metric ($FG$) to quantify performance drops on original benchmarks after fine-tuning. StackLLM consistently demonstrated superior retention in complex cognitive domains.

  • Reading Comprehension (RC): StackLLM showed significantly less forgetting ($FG = 6.8$) compared to the baseline LLM ($FG = 11.9$).
  • Reasoning: StackLLM outperformed the baseline with an $FG$ of $7.3$ versus $8.2$.
  • Domain Knowledge: Interestingly, both models showed negative forgetting (performance gains) in MMLU scores, suggesting that instruction tuning can serve as a refresher for general facts.
Comparison of forgetting (FG) across different task categories for StackLLM 7B and the baseline LLM 7B.

The Bias Paradox

A surprising trade-off emerged regarding model bias. While the baseline LLM saw its bias ratio drop from 63.4% to 55.8% during fine-tuning—a typical de-biasing effect—StackLLM’s bias remained "sticky," moving only slightly from 60.7% to 61.3%. This suggests that the structural redundancy of stacked layers may create internal representations of bias that are more resistant to modification during sequential learning.

Limitations and Future Outlook

While model growth offers clear benefits for knowledge retention, several questions remain. The study was limited to 7B parameter models and a sequence of three tasks; real-world applications involving hundreds of tasks or larger 70B+ models may yield different dynamics. Furthermore, the "why" behind StackLLM's bias persistence requires deeper investigation into whether stacked layers create "pockets" of knowledge that are less accessible to standard fine-tuning gradients. Future work should focus on optimizing fine-tuning schedules to prevent the overfitting observed in early stages while leveraging the retention benefits of growth-based initialization.

Summary updated

BibTeX
@misc{sualp2025mitigating,
  author = {Ege Süalp and Mina Rezaei},
  title = {{Mitigating Catastrophic Forgetting in Continual Learning through Model Growth}},
  year = {2025},
  eprint = {2509.01213},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.01213},
}
Figure