In the study of continual learning, "catastrophic forgetting" is typically treated as an algorithmic failure to be solved with complex memory buffers or regularization constraints. However, a growing body of evidence suggests that a neural network’s fundamental architecture—specifically its width and depth—is a primary determinant of how much it forgets. By shifting the focus from how we train to what we train, researchers have found that simple structural choices can act as powerful implicit regularizers for knowledge retention.
The Stability of Width and Lazy Training
The most consistent finding across this research is that wider neural networks are naturally more resistant to forgetting. As demonstrated by Iman Mirzadeh in Wide Neural Networks Forget Less Catastrophically, increasing the width of a model—even while keeping the total parameter count constant by reducing depth—significantly improves its ability to retain old tasks.
The intuition behind this "width effect" lies in the lazy training regime. In very wide networks, the internal representations (features) do not need to change drastically to minimize the loss on a new task. Instead, the model can rely on its massive, over-parameterized space to find solutions that are nearly orthogonal to previous tasks. Iman Mirzadeh notes that wider models produce sparser and more orthogonal gradients, meaning updates for Task B are less likely to overwrite the weight configurations essential for Task A.
The Fragility of Depth and Representational Drift
While width provides a buffer, depth appears to exacerbate forgetting. Research by Etash Kumar Guha and Vihan Lakshman in On the Diminishing Returns of Width for Continual Learning shows that deeper architectures create more complex transformation paths that are inherently harder to stabilize.
To understand why, consider the "telephone game" of a deep network. If the early layers shift their representations to accommodate a new task, every subsequent layer receives a fundamentally different input signal. This "representational drift" cascades through the network. Joonkyu Kim formalizes this in Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective, proving that deeper layers forget more severely because they are more sensitive to the cumulative shifts of the layers preceding them.
The Trade-off Between Richness and Memory
If width is so beneficial, why not make every model infinitely wide? Jacopo Graldi and Lorenzo Noci address this in The Importance of Being Lazy: Scaling Limits of Continual Learning. They identify a fundamental tension between "rich" feature learning and "lazy" stability.
- Rich Learning: The model learns powerful, task-specific features by significantly moving its weights. This is great for performance on the current task but catastrophic for memory, as it overwrites old features.
- Lazy Learning: The model stays close to its initial state, re-weighting existing features. This is great for memory but may limit the model's ability to learn complex new concepts.
Jacopo Graldi argues that width only helps if it pushes the network toward the lazy regime. However, there is a "sweet spot" for feature learning that balances these two. Interestingly, this optimal balance point appears to transfer across scales, from small models to massive ones.
Diminishing Returns and Local Overfitting
The benefits of width are not infinite. Etash Kumar Guha demonstrates that the gains in knowledge retention eventually plateau, following a law of diminishing returns. Furthermore, even high-capacity models suffer from "local overfitting." As Daphna Weinshall explains in Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation, models often forget how to classify specific sub-regions of data even as their overall accuracy improves.
This suggests that while architecture provides a more stable foundation, forgetting is also a byproduct of how adaptive systems reorganize information. Ben Sanati even suggests in Forgetting is Everywhere that a moderate amount of forgetting might be a "feature" rather than a bug, allowing approximate learners to adapt more efficiently to new information by shedding predictive inconsistencies.
Summary of Intuition
In the context of continual learning, width acts as a stabilizer and depth acts as an amplifier of change. A wide, shallow network is like a large, flat organization where individual changes don't disrupt the whole. A deep, narrow network is like a tall hierarchy where a change at the bottom forces everyone above to change how they work. To build models that remember, we must balance the representational power of depth with the structural stability of width, while navigating the inherent trade-off between learning "rich" new features and preserving the "lazy" status quo.