When a neural network learns a sequence of tasks, it often suffers from catastrophic forgetting—overwriting past knowledge to accommodate new information. Intuitively, one might expect forgetting to scale linearly with task dissimilarity: the more a new task diverges from an old one, the more it should disrupt the network's internal configurations.
However, a growing body of theoretical and empirical research reveals a more complex, counter-intuitive reality. Forgetting does not scale monotonically with task difference. Instead, interference peaks at intermediate levels of task similarity. Tasks that are moderately related to past knowledge cause significantly more disruption than tasks that are either nearly identical or completely orthogonal. Understanding why this "intermediate similarity peak" occurs requires looking at how gradient descent navigates the geometry of representation sharing.
The Three Regimes of Task Similarity
To build intuition for this non-monotonic behavior, we can divide the spectrum of task similarity into three distinct regimes, as characterized by Sebastian Goldt et al. and Vinay Venkatesh Ramasesh et al.:
Forgetting
^
| /\
| / \
| / \
| / \
+---/--------\----> Task Dissimilarity
High Intermediate Low (Orthogonal)
Similarity Similarity Similarity
- High Similarity (The Cooperative Regime): When a new task is highly similar to a previous one, they share an underlying feature structure. The network requires only minimal weight updates to achieve high performance on the new task. Because the existing internal representations are already well-suited for the objective, the original task's performance remains intact.
- Low Similarity / Orthogonality (The Independent Regime): When tasks are completely unrelated, they occupy distinct, orthogonal directions in the data space. In modern, highly overparameterized models, the high-dimensional weight space provides ample degrees of freedom. As Daniel Goldfarb et al. demonstrate analytically, this overparameterization allows the network to find orthogonal subspaces for the new task, learning it safely without overwriting the weight configurations dedicated to the old task.
- Intermediate Similarity (The Conflict Regime): This is the danger zone. The tasks are similar enough to activate the same neural pathways and hidden representations, but different enough to demand conflicting outputs or distinct feature modifications. As a result, the network experiences maximal representational erasure.
The Mechanism: Node Re-Use vs. Node Activation
Why does the network choose to overwrite old representations when facing intermediate tasks, rather than routing around them? Sebastian Lee et al. explain this through the Maslow’s Hammer hypothesis, which frames continual learning as a fundamental trade-off between two optimization strategies:
- Node Re-use: Leveraging existing specialized features to quickly learn the new task (efficient for short-term transfer).
- Node Activation: Allocating and training uncommitted or new nodes to handle the new task (safe for long-term memory).
When tasks are intermediately similar, gradient descent is "tempted" by the efficiency of node re-use. Because the new task correlates partially with existing features, the gradients strongly pull those pre-existing specialized nodes into the service of the new task. However, because the tasks are not identical, adapting these nodes requires modifying their weights significantly. This hybrid approach—re-using a node but altering its core function—directly overwrites the knowledge required for the first task.
This representational conflict is not unique to artificial networks. Exploring these dynamics in both artificial models and human brain imaging, Nicholas Menghi et al. found that intermediate similarity triggers severe cognitive interference. To mitigate this, biological systems must actively "orthogonalize"—or decorrelate—their internal representations to force tasks out of the conflict regime.
Structural Drivers of Non-Monotonic Forgetting
The severity and presence of the intermediate similarity peak depend heavily on the structural properties of the network, particularly its initialization and specialization.
The authors of recent work on network specialization demonstrate that non-monotonic forgetting is intimately tied to how localized a network's representations are. Specifically, Devon Jarvis et al. show that highly specialized networks—where specific neurons are dedicated to specific features—suffer acutely from intermediate similarity peaks because their distinct components are easily hijacked. Conversely, non-specialized networks, which rely on diffuse, distributed representations, tend to exhibit a more monotonic forgetting profile where interference simply increases with task dissimilarity.
Furthermore, the exact sequence of learning matters. By deriving explicit mathematical formulas for forgetting across arbitrary task sequences, Sen Lin et al. proved that while task similarity generally aids generalization, its non-monotonic relationship with memory means that standard training schedules are often suboptimal. To keep a model's overall memory sharp, the authors show that it is mathematically advantageous to introduce highly diverse, dissimilar tasks early in the training pipeline, establishing distinct structural boundaries before intermediate tasks can cause representational collapse.
Key Foundational Work
Maslow's Hammer in Catastrophic Forgetting: Node Re-Use vs. Node Activation (2022) — contributed 5 concept elements to this direction.
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics (2021) — contributed 3 concept elements to this direction.
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021) — contributed 2 concept elements to this direction.