How Model Scale and Architecture Shape Catastrophic Forgetting

75 concepts51 papers1 proposal

When a neural network sequentially learns new tasks, it suffers from catastrophic forgetting—the tendency to abruptly overwrite previously acquired knowledge. For years, the machine learning community has combated this problem primarily through external algorithmic interventions, such as maintaining data replay buffers or applying parameter regularization penalties.

However, an emerging perspective shifts the focus from how we train models to how we build them. A growing body of research demonstrates that a network's structural hyperparameters—specifically its width, depth, and overall parameter scale—fundamentally alter its internal geometry, directly dictating its inherent capacity to retain or erase information.

The Geometry of Width: Lazy Training and Orthogonality

Intuitively, it seems obvious that a larger model should forget less simply because it has "more room" to store information. But the architectural dimension along which a model grows matters immensely. Widening a network (increasing the number of channels or hidden units per layer) acts as a powerful, natural buffer against forgetting.

As demonstrated by Iman Mirzadeh et al. in Architecture Matters in Continual Learning, simple modifications to network width can match or even exceed the performance of specialized continual learning algorithms. To understand why, we must look at the optimization dynamics of wide networks.

The authors of Wide Neural Networks Forget Less Catastrophically reveal that as network width increases, the model enters a "lazy training" regime. In this state, the network's internal features change very little during training, and the gradients of sequential tasks become highly sparse and mutually orthogonal. When the network learns a new task, its weight updates occur along geometric directions that do not intersect with the sub-spaces utilized by previous tasks. Because the gradients are orthogonal, learning Task B does not overwrite the parameter configurations supporting Task A.

Narrow Network (High Overlap):
Task A Gradient:  =======>
Task B Gradient:  <======= (Overwrites Task A)

Wide Network (Orthogonal Subspaces):
Task A Gradient:  =======>
Task B Gradient:  ↑
                  ║ (No interference)

However, this structural shield has clear limits. Etash Kumar Guha demonstrated in On the Diminishing Returns of Width for Continual Learning that the relationship between width and knowledge retention is governed by a power law. As a network grows exceptionally wide, the marginal benefit to memory retention plateaus, making further width scaling computationally inefficient.

Crucially, Jacopo Graldi resolved a major theoretical tension in The Importance of Being Lazy: Scaling Limits of Continual Learning by showing that width only mitigates forgetting if the scaling process successfully pushes the model into this static, lazy regime. If the training parameterization forces the model to maintain active, "rich" feature learning as it grows, the benefits of increased width for knowledge retention completely disappear.

The Peril of Depth: Cascading Discrepancies

If increasing width acts as a stabilizing force, increasing depth (the number of sequential layers) often does the exact opposite. Deeper architectures frequently exacerbate catastrophic forgetting.

To understand why depth destabilizes memory, we can look at how representations shift across layers. Joonkyu Kim et al. introduced an analytical framework in Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective to quantify "representation discrepancy"—the internal drift of hidden layers during sequential training. They proved that representation forgetting is not uniform; rather, it accelerates and becomes more severe in the deeper layers of a network.

Think of a deep neural network as a sequence of cascading coordinate transformations. When a model is updated on a new task, slight parameter adjustments occur in the early layers. As the input signal propagates forward, these minor alterations are compounded non-linearly layer by layer. By the time the signal reaches the deepest layers, the cumulative transformation path has drifted entirely. The deep layers see a completely distorted representation space, destroying the network's ability to execute its original logic.

This phenomenon explains why Etash Kumar Guha observed that deeper architectures create complex transformation paths that are inherently harder to stabilize across non-stationary data streams. Similarly, Bruno Deprez et al. note in Advances in Continual Graph Learning for Anti-Money Laundering Systems: A Comprehensive Review that high-capacity, deep networks can paradoxically accelerate forgetting by overfitting aggressively on the most recent task, rapidly overwriting the fragile, distributed weights associated with historical data.

Pre-training and the Foundation Model Buffer

When we scale models globally—increasing parameters, width, and depth simultaneously within massive architectures like Transformers—the interaction with forgetting changes. In large-scale foundation models, massive parameter counts combined with diverse pre-training create a highly resilient baseline.

Vinay Venkatesh Ramasesh et al. systematically evaluated this in Effect of scale on catastrophic forgetting in neural networks, showing that scaling up both parameter count and pre-training dataset size enhances a network's resistance to forgetting according to a predictable power law.

The underlying intuition rests on the loss landscape. Sanket Vaibhav Mehta et al. discovered in An Empirical Investigation of the Role of Pre-training in Lifelong Learning that diverse pre-training fundamentally changes the geometry of the loss surface, guiding the model into exceptionally "flatter" minima. When a model settles into a flat minimum, its performance is highly robust to small parameter perturbations. When subsequent fine-tuning or domain adaptation shifts the weights, the model remains within the low-loss valley of the original tasks.

Sharp Minimum (Low Scale/No Pre-training):
   \     /
    \ _ /   <-- Small weight shift pushes model up the cliff (Severe Forgetting)
     ▼

Flat Minimum (High Scale + Pre-training):
 \               /
  \___________ /  <-- Small weight shift keeps model in low-loss valley (High Retention)
       ▼

This geometric stability explains why large models, such as those studied by Tu Vu et al. in Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation and Çağatay Yıldız et al. in Investigating Continual Pretraining in Large Language Models: Insights and Implications, can absorb new languages or domains sequentially with far less degradation than smaller models.

The High-Capacity Paradox

Despite the protective effects of scale, massive parameter capacity is not an absolute cure. Instead, it introduces a distinct structural paradox: the very properties that make an architecture highly effective at capturing complex signals can also make it highly efficient at erasing old ones.

Yi Tay demonstrated in DSI++: Updating Transformer Memory with New Documents that scaling up transformer parameter counts does not inherently protect them from abrupt knowledge loss during sequential document indexing, proving that architectural scale cannot entirely substitute for explicit stability mechanisms.

This limitation arises because high-capacity models possess immense semantic plasticity. As observed by Jesús Lovón-Melgarejo et al. in Studying Catastrophic Forgetting in Neural Ranking Models, models that excel at capturing intricate, high-dimensional semantic patterns often exhibit the most severe catastrophic forgetting. Their high plasticity allows them to rapidly reconfigure their internal logic to maximize performance on a new data distribution, resulting in a more aggressive erasure of the parameters associated with historical tasks.

This tension is particularly acute in structured reasoning domains. For instance, Zhuang Li et al. found in Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers that large pre-trained models can exhibit higher rates of catastrophic forgetting in semantic parsing tasks than simpler architectures, as their highly expressive embeddings are easily overwritten during sequential updates.

Ultimately, understanding continual learning requires looking beyond training tricks and examining the structural blueprint of the network itself. Width provides a geometric buffer through gradient orthogonality and lazy dynamics, while depth introduces cascading instabilities. Total scale and pre-training offer a flatter, more resilient loss landscape, but must always be balanced against the volatile plasticity that comes with high capacity.

Key Foundational Work

Does an LSTM forget more than a CNN? An empirical study of catastrophic forgetting in NLP (2019) — contributed 4 concept elements to this direction.
Continual Learning of Object Instances (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_effectiveness_forgetting_trade_off_d6c0bfec,
  title = {effectiveness-forgetting trade-off},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/effectiveness-forgetting-trade-off/rdr_d6c0bfec7a7c4897931e96e2ef0c2141},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_effectiveness_forgetting_trade_off_d6c0bfec_v1,
  title = {effectiveness-forgetting trade-off},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/effectiveness-forgetting-trade-off/rdr_d6c0bfec7a7c4897931e96e2ef0c2141/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake